Conditional Models for Combining Diverse Knowledge Sources in Information Retrieval

March 29, 2006
Rong Yan | Carnegie Mellon University's School of Computer Science

Combining the outputs from multiple retrieval sources/engines is of great importance to a number of retrieval tasks such as multimedia retrieval, web retrieval and meta-search. For example, meta-search attempts to refine retrieval outputs by combining the ranked lists generated from different search engines. Despite the huge amount of combination strategies available, most of them are either completely independent on query topics or dependent on some manually defined query classes. To improve upon this, I first introduce a conditional probabilistic retrieval model as a principled framework for retrieval source combination. Based on this framework, I propose a novel combination approach called probabilistic latent query analysis (pLQA), which can discover latent query classes without prior human knowledge and merge retrieval sources adaptively according to query topics. To further adapt the combination function for individual queries, I also develop the probabilistic local context analysis(pLCA), which can automatically leverage “unlearned” retrieval sources via an undirected graphical model formalism. Experimental results on two large-scale retrieval tasks, i.e., multimedia retrieval and meta-search, demonstrate that the proposed methods can achieve considerable performance gains. Our future work includes extending the proposed methods to other applications such as question answering, cross-lingual IR, multi-engine machine translation, collaborative filtering and so forth.

Speaker Details

Rong Yan is a doctoral candidate at the Language Technologies Institute in Carnegie Mellon University’s School of Computer Science. He obtained his B.E. in computer science from Tsinghua University, Beijing in 2001. His research interests include information retrieval, video content analysis, data mining and machine learning. He received the ACM Multimedia Best Paper Runner-Up award in 2004. He is also the leading architect of the manual video retrieval system that ranks No.1 in TRECVID evaluation ‘03/’05.