{"id":851413,"date":"2022-06-13T09:24:26","date_gmt":"2022-06-13T16:24:26","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-event&p=851413"},"modified":"2022-11-10T22:41:33","modified_gmt":"2022-11-11T06:41:33","slug":"sumeval-2022","status":"publish","type":"msr-event","link":"https:\/\/www.microsoft.com\/en-us\/research\/event\/sumeval-2022\/","title":{"rendered":"SUMEval 2022"},"content":{"rendered":"\n\n\n\n\n
\n
\n

Massively Multilingual Language Models (MMLMs) are trained on around 100 languages of the world, however, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages. The languages present in evaluation benchmarks are usually high-resource and largely belong to the Indo-European language family. This makes current multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. Although efforts are being made to create benchmarks that cover a larger variety of tasks, languages, and language families, it is unlikely that we will be able to build benchmarks covering all languages and tasks. Due to this, there is recent interest in alternate strategies for evaluating MMLMs, including performance prediction and Machine Translation of test data. We believe that this is an important yet relatively unexplored area of research that has the potential to make language technologies accessible to all<\/strong>. The SUMEval workshop will accept submissions on alternate techniques for scaling up multilingual evaluation. In addition, the workshop will also include a shared task on performance prediction.<\/p>\n<\/div>\n\n\n\n

\n

Timeline<\/h4>\n\n\n\n

Dates subject to change and will be updated here as needed.<\/em><\/p>\n\n\n\n

June 28, 2022<\/strong>: Challenge data released
August 1, 2022<\/strong>: Challenge evaluation begins
August 10, 2022<\/strong>: Challenge ends
August 25, 2022<\/strong><\/s> September 23 2022<\/strong>: Workshop paper submission deadline
October 7, 2022<\/strong>: Notification of Acceptance
October 24, 2022<\/strong>: Camera-ready papers due
November 20 2022<\/strong>: SUMEval 2022 Workshop<\/p>\n\n\n\n

<\/p>\n<\/div>\n<\/div>\n\n\n\n

Topics of interest include but are not restricted to:<\/p>\n\n\n\n