SigmaDolphin

Established: August 15, 2015

Overview

Building a computer system to automatically solve math word problems written in natural language. SigmaDolphin is a project initiated in early 2013 at Microsoft Research Asia, with the primary goal of building a computer intelligent system with natural language understanding and reasoning capacities. We focus on the application of automatic problem solving, i.e., automatically solving problems (especially math word problems) written in natural language.

Motivation

Computer programs can complete many tasks much more effectively and efficiently than human beings, such as calculating the product of two large numbers, or finding all occurrences of a string in a long text. However, the performance of computers on many intelligent tasks is still low. For example, in a chatting scenario, computers often generate irrelevant or incorrect responses; and we can easily find ridiculous results in automatic machine translation; and it is still a very challenging task for state-of-the-art computer programs to solve even primary-school-level math word problems.

A well trained person can apparently outperform computers on the above intelligent tasks. The main reason, we believe, lies in our amazing power of natural language understanding and reasoning (especially common sense reasoning) over state-of-the-art artificial intelligent (AI) systems.

Once we are able to empower computers with the ability of natural language understanding and reasoning, we may see significant performance improvements on text processing tasks like question answering and machine translation. In addition, it may open doors to new applications which are impractical so far because of the limited intelligent level of current computer systems.

Present Targeting Scenario: Math Word Problem Solving

We drive our system design and implementation by the scenario of automatically solving math word problems. It is an ideal starting point to our research because of the following reasons:

  1. Math word problems typically contain concise text with clear semantics
  2. They fundamentally cover a wide range of domains (considering the fact that math word problems are often simplified versions of real problems from various domains)
  3. It is relatively easy to measure performance: A lot of problems have been compiled for education purpose; one can often unambiguously determine whether the solutions of a math problem is correct or not.

Research Topics

The following research topics are highly related to our project.

  1. Meaning representation: How to represent the semantic meaning of natural language text
  2. Semantic parsing: How to parse natural language text into structured semantic representation
  3. Reasoning: How to perform reasoning based on the structured semantic representation as well as common sense knowledge

Datasets

Dataset Name Description
number_word_std A problem collection containing 1,878 number word problems, used as evaluation data in our EMNLP’15 paper. Some subsets are included: linear, linear_t2, and linear_t6. Each subset is further divided into a dev set and a test set. Refer to sigmadolphin.datasets.pdf for more details.r more details.
Dolphin18K Dolphin18K is a dataset created for math word problem solving, containing 18,460 problems posted by users on the community question answering site, Yahoo! Answers. Please refer to dolphin18.pdf for more details. Please cite our ACL 2016 paper when you use this dataset in your research.

Research Papers

  • Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu and Yong Rui. 2015. Automatically Solving Number Word Problems by Semantic Parsing and Reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing 2015 (EMNLP 2015), Lisbon, Portugal. [Full-text]
  • Danqing Huang, Shuming Shi, Jian Yin, and Chin-Yew Lin: How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation, ACL 2016.
  • Danqing Huang, Shuming Shi, Jian Yin, and Chin-Yew Lin. 2017. Learning Fine-Grained Expressions to Solve Math Word Problems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing 2017 (EMNLP 2017), Copenhagen, Denmark.

People