Steering and Capturing Human Insight for Large-Scale Learning of Visual Objects

August 23, 2010
Kristen Grauman | University of Texas

An important factor determining our success on object recognition and image search problems is how machine learning algorithms solicit and exploit human knowledge. Existing recognition approaches often manage human supervision in haphazard ways, and only allow a narrow, one-way channel of input from the annotator to the system. We propose algorithms that can steer human insight towards where it will have the most impact, and expand the manner in which recognition methods can assimilate that insight. The underlying goal is to use manual effort cost-effectively for robust visual learning.

More specifically, I will present an approach to actively seek annotators’ input when training an object recognition system. Unlike traditional active learning methods, we target not only the example for which a label is most needed, but also the type of label itself (e.g., an image tag vs. full segmentation). Further, since annotations should be fielded by distributed, uncoordinated annotators, we develop cost-sensitive selection algorithms to compute far-sighted predictions of which batches of data ought to be labeled next. Finally, beyond “asking” annotators the right questions, I will show how we can “listen” more deeply to what image-taggers unknowingly reveal in their annotations, by learning implied cues about object prominence from lists of ordered keywords. Using these cues, we improve state-of-the-art object detection and image retrieval results on benchmark datasets.

This is work done with Sudheendra Vijayanarasimhan and Sung Ju Hwang.

Speaker Details

Kristen Grauman is a Clare Boothe Luce Assistant Professor in the Department of Computer Science at the University of Texas at Austin. Her research in computer vision and machine learning focuses on visual search and object recognition. Before joining UT-Austin in 2007, she received her Ph.D. in the EECS department at MIT, working in the Computer Science and Artificial Intelligence Laboratory. She is a Microsoft Research New Faculty Fellow, and a recipient of an NSF CAREER award and the Howes Scholar Award in Computational Science.