I am a Principal Researcher and manager of the Information and Data Sciences group in Microsoft Research AI. I am interested in the development, improvement, and analysis of machine learning methods with a focus on systems that can aid in the automatic analysis of natural language as components of adaptive systems or information retrieval systems. My current focus is on contextually intelligent assistants. I also maintain an active interest in contextual and personalized search, enriched information retrieval, active sampling and learning, hierarchical and large-scale classification, and human computation and preferences.

My past work has examined a variety of areas — primarily ensemble methods, calibrating classifiers, search query classification and characterization, and redundancy and diversity, but also extending to transfer learning, machine translation, recommender systems, and knowledge bases. In addition to my research, I engage in a variety of professional service activities for the machine learning, data mining, and information retrieval communities.

Before coming to Microsoft, I obtained my Ph.D. from the Computer Science Department at Carnegie Mellon University under Jaime Carbonell and John Lafferty.   Prior to that I worked with Ray Mooney and Robert Causey during my undergraduate days in the Computer Science, Philosophy, and Plan II Honors departments at the University of Texas at Austin. My old CMU resources can be found at the link.


Contextually Intelligent Assistants

A truly intelligent virtual assistant could make many new functions possible. As we envision this agent, it would have the capabilities to: Understand your world, Leverage context to inform its actions, Interact with you naturally and effectively. The Contextually Intelligent Assistants project makes progress toward the type of contextual intelligence needed for next-generation assistants.  It does this by improving the state-of-the-art in understanding task intent from task descriptions; modeling key contextual signals, such as location…

TREC Web Track 2013

Established: October 30, 2012

This page contains guidelines and other information on the TREC 2013 Web Track. For the TREC 2014 Web Track page see http://www.umich.edu/~kevynct/trec-web-2014. TREC 2013 Web Track Guidelines Kevyn Collins-Thompson, Microsoft Research Paul N. Bennett, Microsoft Research Fernando Diaz, Microsoft Research Charles Clarke, University of Waterloo Ellen Voorhees (NIST Contact) NEW: Updated guidelines as of June 6, 2013. NEW: Updated ClueWeb12-related resources as of July 2, 2013. Welcome to the TREC 2013 Web Track. Our goal…




















Professional Service


  • ICML 2009 Tutorial on Machine Learning in IR
  • ECIR 2010 Tutorial on Machine Learning in IR: Recent Successes and New Opportunities

Reviewing and Organizational Service

Journal Editorial Service

  • Guest Editor-in-Chief, ACM Transactions on Information Systems, Special Issue on Contextual Search and Recommendation.

Conference Organization

Senior Program Committees

  • CIKM SPC 2011-2013.
  • HComp 2013 Best Paper Committee.
  • ICML Area Chair, 2015.
  • SIGIR Area Chair, 2013-2015, 2017.
  • SIGKDD SPC 2014-2017.
  • WSDM SPC 2018.

Conference Program Committees

  • AAAI Doctoral Consortium 2008-2010.
  • ACL 2008.
  • CHI 2009.
  • CIKM 2010, CIKM SPC 2011.
  • ECIR 2008-2010.
  • ECML 2008 Workshop on Preference Learning.
  • ECML 2009-2010.
  • EMNLP 2012.
  • HLT-NAACL 2007.
  • HCOMP 2009-2011, 2013-2014.
  • ICML 2010-2013.
  • IIIX 2012.
  • SDM 2013.
  • SIGIR 2004-2011.
  • SIGIR 2011 Crowdsourcing for IR Workshop
  • SIGKDD 2011, 2013.
  • WSDM 2011, 2013, 2015.
  • WWW 2010-2013.

Journal Reviewing

  • Reviewer for the Journal of Artificial Intelligence Research (JAIR).
  • Reviewer for the Journal of the American Society for Information Science and Technology (JASIST).
  • Reviewer for the Journal of Machine Learning Research (JMLR).
  • Reviewer for ACM Transactions on Informations Systems (TOIS).
  • Reviewer for ACM Transactions on Internet Technology (TOIT).
  • Reviewer ACM Transactions on the Web (TWeb).


  • Co-organizer IR Seminar Series, Summer 2004 – Summer 2005.

Dissertation Committees

  • Tobias Schnabel, Cornell University (Advisor: Thorsten Joachims). Computer Science Department. Designing Learning Algorithms and Human Behavior for Interactive Systems.
  • Hyunjoon Jung, University of Texas at Austin (Advisor: Matt Lease). School of Information. Temporal Modeling of Crowd Work Quality for Quality Assurance in Crowdsourcing. Defended, November, 2015.
  • Karthik Raman, Cornell University (Advisor: Thorsten Joachims). Computer Science Department. Machine Learning from Human Preferences and Choices. Defended, June, 2015.
  • Joel Pfeiffer, Purdue University (Advisor: Jennifer Neville). Computer Science Department. Overcoming Uncertainty for Within-Network Statistical Relational Learning. Defended, December 2014.

Current & Former Intern Collaborators

Current & Former Intern Collaborators

Current affiliation (affiliation at time of internship if different)

  • Jan Benetka, PhD Student, Norwegian University of Science and Technology, Computer Science
  • Mustafa Bilgic, Associate Professor, Illinois Institute of Technology (University of Maryland, Computer Science)
  • Fedor Borisyuk, Senior Software Engineer, LinkedIn, (Nizhny Novgorod State University, Computer Science)
  • Horatiu Bota, PhD Student, University of Glasgow, Computer Science
  • Ben Carterette, Associate Professor, Computer & Information Sciences, University of Delaware (University of Massachusetts, Computer Science)
  • Xi Chen, Assistant Professor, NYU Stern (Carnegie Mellon University, Machine Learning Department)
  • Carsten Eickhoff, Postdoctoral Researcher, ETH Zurich (TU Delft, Computer Science)
  • Dhivya Eswaran, PhD Student, Carnegie Mellon University, Computer Science (IIT Madras, Computer Science)
  • Shobeir Fakhraei, Research Scientist, University of Southern California Information Sciences Institute (University of Maryland College Park, Computer Science)
  • Alexander Fishkov, PhD Student, Saint Petersburg Academic University of the Russian Academy of Sciences, Applied Mathematics and Informatics
  • Cristina Garbacea, PhD Student, University of Michigan, School of Information (University of Amsterdam, Computer Science)
  • David Graus, Data Scientist, FD Mediagroup (University of Amsterdam, Computer Science)
  • Ahmed Hefny, PhD Student, Carnegie Mellon University, Machine Learning Department
  • Jagadeesh Jagarlamudi, Software Engineer, Google (University of Maryland, Computer Science)
  • Ashiqur KhudaBukhsh, PhD Student, Carnegie Mellon University, Computer Science Department
  • Jin Kim, Data Scientist, Snap (University of Massachusetts, Computer Science)
  • Alex Kotov, Assistant Professor, Wayne State University, Computer Science (University of Illinois-UC, Computer Science)
  • Danai Koutra, Assistant Professor, University of Michigan, Computer Science & Engineering (Carnegie Mellon University, Computer Science)
  • Edith Law, Assistant Professor, University of Waterloo, Computer Science (Carnegie Mellon University, Machine Learning Department)
  • Jay-Yoon Lee, PhD Student, Carnegie Mellon University, Computer Science Department
  • Rishabh Mehrotra, PhD Student, University College London, Computer Science Department
  • Nam Nguyen, Software Engineer, Facebook (Cornell University, Computer Science)
  • Jason Portenoy, PhD Student, University of Washington, Information School
  • Joel Pfeiffer, Applied Researcher, Microsoft (Purdue University, Computer Science)
  • Kira Radinsky, Chief Scientist (IL) & Director of Data Science, eBay (Technion, Israel Institute of Technology)
  • Karthik Raman, Research Scientist, Google (Cornell University, Computer Science)
  • Xin Rong, PhD Student, University of Michigan, School of Information
  • Tobias Schnabel, PhD Student, Cornell University, Computer Science Department
  • Marc Sloan, Co-founder & CEO of Context Scout (University College London, Computer Science)
  • Lidan Wang, Research Scientist, IBM T.J. Watson (University of Maryland, Computer Science)
  • Yue Wang, PhD Student, EE & CS Department
  • Rongjing Xiang, Software Engineer, Google (Purdue University, Computer Science)
  • Qian Zhao, PhD Student, University of Minnesota, Computer Science