- Dialog management: Applications of machine learning; integration of expert knowledge; multi-modal dialog management; planning techniques; on-line improvement.
- Dialog and user modeling: Tracking and quantifying uncertainty in dialog state for human/computer dialog; representational structures for dialog state; ontology integration; simulation.
- Turn-taking: Use of continuous/incremental speech recognition for turn-taking; integration of conversation history and user model for turn-taking.
- Confidence scoring: Machine-learning-based approaches to confidence scoring; calibration in confidence scoring; features for confidence scoring.
- Planning under uncertainty: Markov decision processes (MDPs); partially observable Markov decision processes (POMDPs); reinforcement learning.
- Deep learning: Methods for modeling dialog and implementing dialog control using deep learning, particularly recurrent neural networks.
- Hybrid Code Networks: End-to-end learning for dialog systems that support domain knowledge in the form of software (code) and action templates. [video, paper, demo description]
- Language Understanding Intelligent Service (LUIS): LUIS is a new, fast way of building language understanding models. LUIS is available at luis.ai.
- Dialog Systems Technology Challenge (DSTC — originally Dialog State Tracking Challenge): An on-going series of research community challenge tasks related to dialog systems and dialog state tracking. DSTC homepage.
- Microsoft Research, Principal Researcher, 2012-Present
- AT&T Labs Research, Principal Member of Technical Staff, 2006-2012
- Cambridge University, Ph D, Engineering Dept, 2002-2006
- Edify Corp, Senior consultant – Usability and Speech Technology, 2002-2005
- Tellme Networks, Voice Application Development Manager, 2000-2001
- McKinsey & Company, Associate, 1999-2000
- Cambridge University, Masters, Speech/Language Processing, 1998-1999
- Princeton University, BSE, Electrical Engineering, 1994-1998
- Microsoft "high-potential talent" designation: 2014-2017
- SigDial best paper award: 2014
- IEEE SLT Best Poster Presentation in session: 2014
- SigDial Best student paper award: 2011
- ISCA award for best paper in Computer Speech and Language, for the period 2005-2009: 2010
- AT&T Key Contributor award: 2010, 2009
- AT&T Intellectual Property achievement award: 2010
- AT&T Labs President Excellence Award: 2010
- AT&T Labs Research excellence award: 2007
- AVIOS Best paper award: 2003
Interns (current and former)
Kavosh Asadi, Summer 2016 and 2017.
Reinforcement learning with recurrent neural networks for dialog systems.
Currently a PhD student with Michael Littman, Brown University.
Hao Fang, Summer 2015.
Natural language understanding.
Currently a PhD student with Mari Ostendorf, University of Washington.
Nobal Niraula, Summer 2014.
Natural language understanding.
Currently with Boeing Research & Technology.
Pradeep Dasigi, Summer 2014.
Currently a PhD student with Ed Hovy, CMU.
Ke Zhai, Summer 2013.
Currently an Applied Scientist at Microsoft.
He He, Summer 2013.
Action selection in dialog systems (co-mentored with Lihong Li as lead mentor).
Currently a post-doc researcher with Percy Liang at Stanford University.
Angeliki Metallinou, Summer 2012.
Belief tracking in dialog systems (co-mentored with Dan Bohus).
Currently a Speech Scientist at Amazon.
Ethan Selfridge, Summer 2010 and 2011 at AT&T.
Turn-taking in dialog systems.
Currently at Interactions Corp.
Hamid Chinaei, Summer 2010 at AT&T.
Reinforcement learning for dialog systems.
Currently with Nuance Communications.
John Asmuth, Summer 2009 at AT&T.
Bayesian approaches to reinforcement learning.
Currently at Google.
Lihong Li, Summer 2008 at AT&T.
Feature selection in reinforcement learning.
Currently a Researcher with Microsoft Research.
Umar Syed, Summer 2007 at AT&T.
Learning user models from unlabeled data.
Currently a research scientist at Google.
8010364: System and method for applying probability distribution models to dialog systems in the troubleshooting domain
8140328: User intention based on N-best list of recognition hypotheses for utterances in a dialog
8433578: System and method for automatically generating a dialog manager
8457968: System and method for efficient tracking of multiple dialog states with incremental recombination
8473292: System and method for generating user models from transcribed dialogs
8660844: System and method of evaluating user simulations in a spoken dialog system with a diversion metric
8914288: System and method for advanced turn-taking for interactive spoken dialog systems
8954319: System and method for generating manually designed and automatically optimized spoken dialog systems
9015048: Incremental speech recognition for dialog systems
9547471: Generating computer responses to social conversational inputs
[ + 8 more patent applications in progress with the US Patent Office ]
Appointments, editor, and organizer roles
- IEEE Workshop on Speech and Language Technology (SLT), Advisory Board, 2016.
- Dialogue and Discourse: Special Issue on Dialogue State Tracking, Co-lead guest editor, 2015.
- IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Co-general chair, 2015.
- IEEE Speech and Language Technical Committee (SLTC), Elected member, 2015-2017; 2009-2011.
- Dialog State Tracking Challenge / Dialog System Techology Challenge (DSTC): Organizer DSTC1-DSTC5; Steering committee DSTC6
- NAACL, Area chair, Dialogue and Interactive Systems, 2015.
- SIGDIAL – Special Interest Group on Discourse and Dialogue, President, 2017-2018.
- SIGDIAL – Special Interest Group on Discourse and Dialogue, Vice-President, 2013-2017.
- IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Technical Chair (Dialog area), 2013.
- SIGDIAL Conference 2013, Program committee Co-chair.
- Real Dialog Data Challenge, Advisory board, 2013-2014.
- IEEE Journal of Selected Topics in Signal Processing: Special Issue on Advances in Spoken Dialogue Systems and Mobile Interface (vol 6, issue 8), Co-lead guest editor, 2012.
- IEEE Workshop on Spoken Language Technologies (SLT), Organizing committee, co-finance chair, 2012.
- SIGDIAL Conference, Organizing committee, sponsorship chair, 2012.
- AVIxD / IxDA Workshop: Designing for multi-modal interaction, Co-chair, organizing committee, 2012.
- SIGDIAL Conference, Organizing committee, sponsorship chair, 2011.
- Association for Voice Interaction Design (AVIxD), Board of directors, Treasurer, 2010-2012.
- IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Organizing committee, publicity chair, 2011.
- SIGDIAL Conference, mentoring program coordinator, 2010.
- ICAPS Workshop: POMDP Practitioners Workshop: solving real-world POMDP problems, Organizing committee, 2010.
- Spoken Dialogue Challenge, Organizing Committee: 2010-2012.
- IEEE SLTC Newsletter, Editor-in-chief, 2009-2011.
- SIGDIAL – Special Interest Group on Discourse and Dialogue, Science Advisory Committee, 2007-2012.
- AAAI Workshop on statistical and empirical methods in spoken dialogue systems, organizer, 2006.
- NAACL-HLT Demonstration program co-chair, 2007.
- Young researchers' roundtable on spoken dialogue systems, organizer, 2005.
Journal reviewing and grant panelist
- Transactions of the ACL (TACL) reviewer: 2016, 2017.
- IEEE Transactions on Audio, Speech, and Language Processing reviewer: 2014, 2013, 2012, 2011, 2010, 2008, 2007.
- Computer Speech and Language reviewer: 2012, 2011, 2010, 2009, 2008.
- Computational Linguistics reviewer: 2009.
- Speech Communication reviewer: 2016, 2015, 2014, 2012, 2011, 2010, 2008.
- Journal of Natural Language Engineering reviewer: 2008.
- ACM Transactions on Speech and Language Processing: 2012, 2011, 2010.
- Pattern Recognition Letters reviewer: 2010.
- IEEE Signal Processing Magazine reviewer: 2007.
- Transactions on Systems, Man, and Cybernetics – Part B: Cybernetics reviewer: 2012.
- User Modeling and User-Adapted Interaction (UMUAI): 2015.
- Service on 6 grant review panels, for US and international funding agencies.
Conference program committees
- AAAI program committee: 2018.
- ACL program committee: 2017, 2016, 2015, 2014, 2013, 2012, 2011.
- ASRU program committee: 2017, 2013, 2011, 2009, 2007.
- CHI reviewer: 2018, 2013.
- COLING Program committee: 2016, 2014.
- EACL scientific review committee: 2017, 2014, 2012, 2009.
- EMNLP program committee: 2017, 2016, 2015, 2014, 2013, 2009, 2008, 2006
- EUSIPCO reviewer: 2015.
- ICASSP reviewer: 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007.
- ICMI reviewer: 2017, 2016, 2015, 2014.
- IJCAI senior program committee: 2015.
- IJCAI program committee: 2011, 2009, 2007.
- IJCNLP program committee: 2017.
- INTERSPEECH Scientific Review Committee: 2017, 2016, 2015, 2014, 2013, 2012, 2011.
- IUI reviewer: 2015, 2014, 2013.
- IWSDS program committee: 2018, 2017, 2016, 2015, 2014.
- LREC scientific committee: 2018.
- NAACL reviewer: 2012, 2009, 2007.
- NIPS reviewer: 2015, 2010.
- SIGDIAL program committee: 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007.
- SLT program committee: 2016, 2014, 2012, 2010, 2008.
Workshop and other program committees
- ACL demo committee: 2011.
- ACL Student Research workshop program committee: 2014, 2015.
- COLING Demonstration session: 2016, 2014.
- COLING Workshop: Spoken language technologies for pervasive speech-based and multimodal applications program committee: 2014, 2008
- Workshop on Speech-centric Natural Language Processing, program committee: 2017 (EMNLP), 2016 (COLING)
- DSTC workshop, reviewer: 2017.
- Domain Adaptation for Dialog Agents Workshop (DADA): 2016
- EACL 2014 Workshop on Dialog in Motion: 2014.
- ECAI/IJCAI/AAAI Workshop on Machine Learning for Interactive Systems (MLIS): 2015, 2014, 2013, 2012
- EMNLP Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses: 2014.
- Future and Emerging Trends in Language Technologies (FELT): 2015, 2016.
- ICML Workshop on Learning to Generate Natural Language: 2017.
- IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems program committee: 2011, 2009.
- Machine Learning for Spoken Language Understanding and Interactions Workshop (at NIPS): 2015.
- NAACL Student Research workshop program committee: 2015.
- NIPS Workshop on Learning Semantics: 2014.
- NIPS Conversational Intelligence Challenge: 2017.
- RE-WOCHAT (Workshop on Collecting and Generating Resources for Chatbots and Conversational Agents – Development and Evaluation): 2016 (1), 2016 (2)
- SLAM: Joint ISCA/IEEE International Workshop on Speech, Language and Audio in Multimedia: 2013.
- Workshop on Future and Emerging Trends in Language Technology: 2015.
- Workshop on Future directions and needs in the Spoken Dialog Community: Tools and Data scientific committee: 2012.
- Workshop on Speech-Centric Natural Language Processing: 2017
- Young researchers' roundtable on spoken dialogue systems advisory board: 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2006.
Language Understanding Intelligent Service (LUIS)
LUIS is a new, fast way of building language understanding models. LUIS is available at luis.ai.
Dialog State Tracking Challenge
The Dialog State Tracking Challenge (DSTC) is a research community challenge task for accurately estimating a user’s goal in a spoken dialog system. DSTC homepage.