{"id":693573,"date":"2020-09-29T09:13:18","date_gmt":"2020-09-29T16:13:18","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=693573"},"modified":"2020-10-08T10:18:47","modified_gmt":"2020-10-08T17:18:47","slug":"codexglue-a-benchmark-dataset-and-open-challenge-for-code-intelligence","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/codexglue-a-benchmark-dataset-and-open-challenge-for-code-intelligence\/","title":{"rendered":"CodeXGLUE: A benchmark dataset and open challenge for code intelligence"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

According to Evans Data Corporation (opens in new tab)<\/span><\/a>, there are 23.9 million professional developers in 2019, and the population is expected to reach 28.7 million in 2024. With the growing population of developers, code intelligence, which aims to leverage AI to help software developers improve the productivity of the development process, is growing increasingly important in both communities of software engineering and artificial intelligence.<\/p>\n\n\n\n

When developers want to find code written by others with the same intent, code search (opens in new tab)<\/span><\/a> systems can help automatically retrieve semantically relevant code given natural language queries. When developers are confused about what to write next, code completion (opens in new tab)<\/span><\/a> systems can help by automatically completing the following tokens given the context of the edits being made. When developers want to implement Java code with the same function of some existing body of Python code, code-to-code translation (opens in new tab)<\/span><\/a> systems can help translate from one programming language (Python) to another (Java).<\/p>\n\n\n\n

Code intelligence therefore plays a vital role in Microsoft\u2019s mission to empower developers. As highlighted by Microsoft CEO Satya Nadella at Microsoft Build 2020 (opens in new tab)<\/span><\/a>, the role of developers is more important than ever. GitHub is increasingly the default home for source code, and Visual Studio Code is one of the most popular code editors. Microsoft offers a complete toolchain for developers, bringing together the best of GitHub, Visual Studio, and Microsoft Azure to help developers to go from idea to code and code to cloud.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tSpotlight: Event Series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Research\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Microsoft Research Forum<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Join us for a continuous exchange of ideas about research in the era of general AI. Watch the first four episodes on demand.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tWatch on-demand\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Recent years have seen a surge of applying of statistical models, including neural nets, to code intelligence tasks. Very recently, pre-trained models learned from big programming language data have been inspired by the great success of large pre-trained models like BERT (opens in new tab)<\/span><\/a> and GPT (opens in new tab)<\/span><\/a> in natural language processing (NLP). These models, including IntelliCode (opens in new tab)<\/span><\/a> and CodeBERT (opens in new tab)<\/span><\/a>, obtain further improvements on code understanding and generation problems. However, the area of code intelligence lacks a benchmark suite that covers a wide range of tasks. We have seen that a diversified benchmark dataset is significant for the growth of an area of applied AI research, like ImageNet (opens in new tab)<\/span><\/a> for computer vision and GLUE (opens in new tab)<\/span><\/a> for NLP.<\/p>\n\n\n\n

To address this, researchers from Microsoft Research Asia (Natural Language Computing Group<\/a>) working together with Developer Division and Bing introduce CodeXGLUE (opens in new tab)<\/span><\/a>, a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for code<\/em><\/strong>. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios:<\/p>\n\n\n\n