{"id":293411,"date":"2016-09-15T17:53:03","date_gmt":"2016-09-16T00:53:03","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-event&p=293411"},"modified":"2025-08-06T11:59:15","modified_gmt":"2025-08-06T18:59:15","slug":"dialog-state-tracking-challenge","status":"publish","type":"msr-event","link":"https:\/\/www.microsoft.com\/en-us\/research\/event\/dialog-state-tracking-challenge\/","title":{"rendered":"2011 Dialog System Technology Challenge (DSTC)"},"content":{"rendered":"\n\n

The 2011 Dialog System Technology Challenge — formerly the Dialog State Tracking Challenge — is a series of research community challenge tasks for accurately estimating a user’s goal in a spoken dialog system.Opens in a new tab<\/span><\/p>\n

The Dialog State Tracking Challenge (DSTC) is an on-going series of research community challenge tasks. Each task released dialog data labeled with dialog state information, such as the user’s desired restaurant search query given all of the dialog history up to the current turn. The challenge is to create a “tracker” that can predict the dialog state for new dialogs. In each challenge, trackers are evaluated using held-out dialog data.<\/p>\n

You can find information on the current DTSC8 competition here (opens in new tab)<\/span><\/a>.<\/p>\n

DSTC1<\/h3>\n

DSTC1 used human-computer dialogs in the bus timetable domain. Results were presented in a special session at SIGDIAL 2013 (opens in new tab)<\/span><\/a>. DSTC1 was organized by Jason D. Williams, Alan Black, Deepak Ramachandran, Antoine Raux.<\/p>\n

For DSTC1 data, and more information, see “DSTC1” tabs above.<\/b><\/p>\n

DSTC2 and DSTC3<\/h3>\n

DSTC2\/3 used human-computer dialogs in the restaurant information domain. Results were presented in special sessions at SIGDIAL 2014 (opens in new tab)<\/span><\/a> and IEEE SLT 2014 (opens in new tab)<\/span><\/a>. DSTC2 and 3 were organized by Matthew Henderson, Blaise Thomson, and Jason D. Williams.<\/p>\n

For DSTC2 and DSTC3 data, and more information, see the DSTC2\/3 website. (opens in new tab)<\/span><\/a><\/b><\/p>\n

DSTC4<\/h3>\n

DSTC4 used human-human dialogs in the tourist information domain. Results were presented at IWSDS 2015 (opens in new tab)<\/span><\/a>. DSTC4 was organized by Seokhwan Kim, Luis F. D’Haro, Rafael E Banchs, Matthew Henderson, and Jason D. Williams.<\/p>\n

For more information about DSTC4, see the DSTC4 website. (opens in new tab)<\/span><\/a><\/b><\/p>\n

DSTC5<\/h3>\n

DSTC5 used human-human dialogs in the tourist information domain, where training dialogs were provided in one language, and test dialogs were in a different language. Results were presented in a special session at IEEE SLT 2016 (opens in new tab)<\/span><\/a>. DSTC5 was organized by Seokhwan Kim, Luis F. D’Haro, Rafael E Banchs, Matthew Henderson, Jason D. Williams, and Koichiro Yoshino.<\/p>\n

For more information about DSTC5, see the DSTC5 website. (opens in new tab)<\/span><\/a><\/b><\/p>\n

DSTC6<\/h3>\n

DSTC6 consisted of 3 parallel tracks: End-to-End Goal Oriented Dialog Learning, End-to-End Conversation Modeling, and Dialogue Breakdown Detection. Results will be presented at a workshop immediately after NIPS 2017. DSTC6 is organized by Chiori Hori, Julien Perez, Koichiro Yoshino, and Seokhwan Kim. Tracks were organized by Y-Lan Boureau, Antoine Bordes, Julien Perez, Ryuichi Higashinaka, Chiori Hori, and Takaaki Hori.<\/p>\n

For more information about DSTC6, see the DSTC6 website. (opens in new tab)<\/span><\/a><\/b>Opens in a new tab<\/span><\/p>\n

To join the mailing list, please go to the current DTSC website (opens in new tab)<\/span><\/a>.<\/p>\n

 Opens in a new tab<\/span><\/p>\n

Summary of DSTC1, 2, and 3<\/h3>\n

The Dialog State Tracking Challenge Series: A Review<\/a>. Jason D. Williams, Antoine Raux, and Matthew Henderson. Dialogue & Discourse, April 2016.<\/p>\n

DSTC1<\/h3>\n

The Dialog State Tracking Challenge<\/a>. Jason D. Williams, Antoine Raux, Deepak Ramachadran, and Alan Black. Proceedings of the SIGDIAL 2013 Conference, Metz, France, August 2013.<\/p>\n

DSTC2<\/h3>\n

The Second Dialog State Tracking Challenge<\/a>. Matthew Henderson, Blaise Thomson, and Jason D. Williams. Proceedings of SIGDIAL 2014 Conference, Philadelphia, USA, June 2014.<\/p>\n

DSTC3<\/h3>\n

The Third Dialog State Tracking Challenge<\/a>. Matthew Henderson, Blaise Thomson, and Jason D. Williams. Proceedings IEEE Spoken Language Technology Workshop (SLT), South Lake Tahoe, USA, December 2014.<\/p>\n

DSTC4<\/h3>\n

The Fourth Dialog State Tracking Challenge<\/a>. Seokhwan Kim, Luis F. D’Haro, Rafael E Banchs, Matthew Henderson, and Jason D. Williams. Proceedings IEEE Spoken Language Technology Workshop (SLT), South Lake Tahoe, USA, December 2014.<\/p>\n

DSTC5<\/h3>\n

The Fifth Dialog State Tracking Challenge. Seokhwan Kim, Luis F. D’Haro, Rafael E Banchs, Matthew Henderson, Jason D. Williams, and Koichiro Yoshino. Proceedings IEEE Spoken Language Technology Workshop (SLT), San Diego, USA, December 2016.Opens in a new tab<\/span><\/p>\n

Background and motivation<\/b><\/p>\n

In dialog systems, “state tracking” \u2013 sometimes also called “belief tracking” \u2013 refers to accurately estimating the user\u2019s goal as a dialog progresses. Accurate state tracking is desirable because it provides robustness to errors in speech recognition, and helps reduce ambiguity inherent in language within a temporal process like dialog. Dialog state tracking is an important problem for both traditional uni-modal dialog systems, as well as speech-enabled multi-modal dialog systems on mobile devices, on tablet computers, and in automobiles.<\/p>\n

Recently, a host of models have been proposed for dialog state tracking. However, comparisons among models are rare, and different research groups use different data from disparate domains. Moreover, there is currently no common dataset which enables off-line dialog state tracking experiments, so newcomers to the area must first collect dialog data, which is expensive and time-consuming, or resort to simulated dialog data, which can be unreliable. All of these issues hinder advancing the state-of-the-art.<\/p>\n

Challenge format<\/b><\/p>\n

In this challenge, participants will use a provided set of labeled dialogs to develop a dialog state tracking algorithm. Algorithm will then be evaluated on a common set of held-out dialogs, to enable comparisons.<\/p>\n

The data for this challenge will be taken from the Spoken Dialog Challenge (Black et al, 2011), which consists of human\/machine spoken dialogs with real users (not usability subjects). Before the start of the challenge, a draft of the labeling guide and evaluation measures will be published, and comments will be invited from the community. The organizers will then perform the labeling.<\/p>\n

At the start of the challenge \u2013 the development phase \u2013 participants will be provided with a training set of transcribed and labeled dialogs. Participants will also be given code that implements the evaluation measurements. Participants will then have several months to optimize their algorithms.<\/p>\n

At the end of the challenge, participants will be given an untranscribed and unlabeled test set, and a short period to run their algorithm against the test set. Participants will submit their algorithms\u2019 output to the organizers, who will then perform the evaluation. After the challenge, the test set transcriptions and labels will be made public.<\/p>\n

Results of the evaluation will be reported to the community. In this challenge, the identities of participants will not be made public in any written results by the organizers or participants, except that participants may identify themselves (only) in their own written results.<\/p>\n

Schedule<\/b><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
1 July 2012<\/td>\nBeginning of comment period on labeling and evaluation metrics<\/td>\n<\/tr>\n
4-6 July 2012<\/td>\nYRRSDS and SigDial in Korea: announcement of comment period<\/td>\n<\/tr>\n
17 August 2012<\/td>\nEnd of comment period on labeling and evaluation metrics<\/td>\n<\/tr>\n
31 August 2012<\/td>\nEvaluation metrics and labeling guide published; labeling begins<\/td>\n<\/tr>\n
10 December 2012<\/td>\nLabeling ends; data available; challenge begins (14 weeks)<\/td>\n<\/tr>\n
3-5 December 2012<\/td>\nIEEE SLT 2012 (opens in new tab)<\/span><\/a> (Miami): Information session, 5 Dec 3 PM<\/td>\n<\/tr>\n
22 March 2013<\/td>\nFinal system due; evaluation begins (1 week)<\/td>\n<\/tr>\n
29 March 2013<\/td>\nEvaluation output due to organizers<\/td>\n<\/tr>\n
5 April 2013<\/td>\nResults sent to teams<\/td>\n<\/tr>\n
3 May 2013<\/td>\nSigDial paper deadline; write up (4 weeks)<\/td>\n<\/tr>\n
August 2013<\/td>\nSigDial 2013<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Organizers<\/b><\/p>\n

Jason D. Williams<\/a>, Microsoft Research, USA (chair)
\nAlan Black, Carnegie Mellon University, USA
\nDeepak Ramachandran, Honda Research Institute, USA
\nAntoine Raux, Honda Research Institute, USA<\/p>\n

Advisory committee<\/b><\/p>\n

Daniel Boies, Microsoft, Canada
\nPaul Crook, Microsoft, USA
\nMaxine Eskenazi, Carnegie Mellon University, USA
\nMilica Gasic, University of Cambridge, UK
\nDilek Hakkani-Tur, Microsoft, USA
\nHelen Hastie, Heriot Watt University, UK
\nKee-Eung Kim, KAIST, Korea
\nIan Lane, Carnegie Mellon University, USA
\nSungjin Lee, Carnegie Mellon University, USA
\nTeruhisa Misu, NICT, Japan
\nOlivier Pietquin, SUPELEC, France
\nJoelle Pineau, McGill University, Canada
\nBlaise Thomson, University of Cambridge, UK
\nDavid Traum, USC Institute for Creative Technologies, USA
\nLuke Zettlemoyer, University of Washington, USA<\/p>\n

More information<\/b><\/p>\n

The Dialog State Tracking Challenge<\/a>. Jason D. Williams, Antoine Raux, Deepak Ramachadran, and Alan Black. Proceedings of the SIGDIAL 2013 Conference, Metz, France, August 2013.<\/p>\n

Dialog state tracking challenge handbook<\/a>. Antoine Raux, Deepak Ramachandran, Alan Black, and Jason D. Williams. Technical Report.<\/p>\n

A belief tracking challenge task for spoken dialog systems<\/a>. Jason D. Williams. Proceedings NAACL HLT 2012 Workshop on Future directions and needs in the Spoken Dialog Community: Tools and Data, June 2012.<\/p>\n

Spoken Dialog Challenge 2010: Comparison of Live and Control Test Results<\/a>. Alan W Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, Jason D. Williams, Kai Yu, Steve Young, and Maxine Eskenazi. Proceedings SIGDIAL, Portland, Oregon, USA, 2011.Opens in a new tab<\/span><\/p>\n

Training data<\/b><\/p>\n