Learning using Large Datasets

December 10, 2007
Leon Bottou | NEC Labs America

The statistical learning theory suggests to choose large capacity models that barely avoid over-fitting the training data. In that perspective, all datasets are small. Things become more complicated when one considers the computational cost of processing large datasets.

Computationally challenging training sets appear when one want to emulate intelligence: biological brains learn quite efficiently from the continuous streams of perceptual data generated by our six senses, using limited amounts of sugar as a source of power. Computationally challenging training sets also appear when one want to analyze the masses of data that describe the life of our computerized society. The more data we understand, the more we enjoy competitive advantages.

The first part of the presentation clarifies the relation between the statistical efficiency, the design of learning algorithms and their computational cost. The second part makes a detailed exploration of specific learning algorithms and of their implementation, with both simple and complex examples. The third part considers algorithms that learn with a single pass over the data. Certain algorithms have optimal properties but are often too costly. Workarounds are discussed.

Speaker Details

Léon Bottou received a Diplôme from Ecole Polytechnique, Paris in 1987, a Magistère en Mathématiques Fondamentales et Appliquées et Informatiques from Ecole Normale Supérieure, Paris in 1988, and a PhD in Computer Science from Université de Paris-Sud in 1991. He joined AT&T Bell Labs from 1991 to 1992 and AT&T Labs from 1995 to 2002. Between 1992 and 1995 he was chairman of Neuristique in Paris, a small company pionneering machine learning for data mining applications. He has been with NEC Labs America in Princeton since 2002. Léon’s primary research interest is machine learning. His contributions to this field address theory, algorithms and large scale applications.Léon’s secondary research interest is data compression and coding. His best known contribution in this field is the DjVu document compression technology (http://www.djvuzone.org). Léon is serving on the boards of the Journal of Machine Learning Research and of IEEE Trans. PAMI. He also serves on the scientific advisory board of Kxen Inc (http://www.kxen.com). He won the New York Academy of Sciences Blavatnik Award for Young Scientists in 2007.