Learning Web-based Procedures by Reasoning over Explanations and Demonstrations in Context

Association for Computational Linguistics |

We explore learning web-based tasks from a human teacher through natural language explanations and a single demonstration. Our approach investigates a new direction for semantic parsing that models explaining a demonstration in a context, rather than mapping explanations to demonstrations. By leveraging the idea of inverse semantics from program synthesis to reason backwards from observed demonstrations, we ensure that parsed interpretations are consistent with executable actions in any context, thus simplifying the problem of search during parsing. We present a dataset of explanations paired with demonstrations for web-based tasks (.zip file, 6 mb). Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average) and are competitive with simple exploration-and-demonstration based methods, while requiring no exploration of the environment. In learning to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interesting example of pragmatic language acquisition from grounded contexts without any linguistic annotation.

Publication Downloads

Web Demonstration and Explanation Dataset

May 8, 2020

Web Demonstration and Explanation Dataset: This data was collected for and used in our ACL 2020 paper that demonstrates the potential to effectively combine explanations and demonstrations to learn web-based procedures. This data consists of 520 explanations and corresponding demonstrations of web-based tasks from the Mini Word-of-Bits.

Download Data