How to learn an object recognition system with millions of parameters from a small number of training examples

November 25, 2009
Geoff Hinton | University of Toronto

The human object recognition system has about a trillion synapses and computer vision systems will probably need to learn a similar number of parameters in order to be competitive. This makes it necessary to learn useful feature detectors from unlabeled examples (as the cortex does). I will describe how this can be done and will illustrate the approach with two multi-layer neural networks that perform object recognition on two very different databases. The NORB database has stereo, gray-level images of 5 different object classes with a wide range of viewpoints and lighting conditions. The TINY IMAGES database has 80 million small color images from about 50,000 classes harvested from the web. In both databases the number of labeled examples is far less than a million. Both systems learn about 100 million parameters in a few GPU days and have state-of-the-art performance. I will also show that the same approach works even if the labels are so unreliable that they are mostly wrong. The neural networks I will describe were recently developed by Vinod Nair, Alex Krizhevsky and Marc’Aurelio Ranzato.

Speaker Details

Geoff Hinton is a fellow of the Canadian Institute for Advanced Research and Professor of Department of Computer Science at the University of Toronto. He spent three years from 1998 until 2001 setting up the Gatsby Computational Neuroscience Unit at University College London and then returned to the University of Toronto where he is a University Professor. He is the director of the program on “Neural Computation and Adaptive Perception” which is funded by the Canadian Institute for Advanced Research. Geoffrey Hinton is a fellow of the Royal Society and an honorary foreign member of the American Academy of Arts and Sciences. He was one of the researchers who introduced the back-propagation algorithm that has been widely used for practical applications. His other contributions to neural network research include Boltzmann machines, distributed representations, time-delay neural nets, mixtures of experts, Helmholtz machines and products of experts. More recently, he and his collaborators have introduced efficient, unsupervised, learning procedures for deep networks containing many layers of non-linear features. They have demonstrated that, in addition to creating excellent generative models, this unsupervised learning greatly improves classification performance in a variety of domains.