Learning to Understand Natural Language in Physically-Grounded Environments

  • Jayant Krishnamurthy | Carnegie Mellon University

We all want computers to understand natural language. Whether it be to command a robot, or answer a question by reading the web, language understanding is a fundamental problem for natural language processing. Physically-grounded settings are an important special case of this problem, with applications in robotics and interactions with embodied systems.

This talk presents Logical Semantics with Perception (LSP), a model for understanding natural language statements within a physically-grounded environment. For example, given an image, LSP can map a description such as “the blue mug to the left of the monitor,” to a set of image segments containing blue mugs left of monitors. Importantly, LSP can be trained directly from natural language / object pairs, which is a natural form of supervision that can be easily obtained from human interaction. I will present experiments applying LSP to several domains, including image understanding (using Microsoft Kinect data) and geographical question answering.

The talk may also include some additional related work on semantic parsing at web-scale for information extraction and question answering. (time permitting)

Speaker Details

Jayant Krishnamurthy is a Ph.D. student in the Computer Science Department at Carnegie Mellon University. Prior to attending Carnegie Mellon, he received an M.Eng and S.B. from the Massachusetts Institute of Technology. Jayant’s research is on machine learning and natural language processing, with a focus on understanding the semantics of natural language. His work is part of the Never-Ending Language Learner (NELL) project at Carnegie Mellon, directed by Tom Mitchell.

    • Portrait of Jeff Running

      Jeff Running

Series: Microsoft Research Talks