Both Sides Now: Generating and Understanding Visually-Grounded Language

  • Peter Anderson | Georgia Tech

From robots to cars, virtual assistants and voice-controlled drones, computing devices are increasingly expected to communicate naturally with people and to understand the visual context in which they operate. In this talk, I will present our latest work on generating and comprehending visually-grounded language. First, we will discuss the challenging task of describing an image (image captioning). I will introduce captioning models that leverage multiple data sources, including object detection datasets and unaligned text corpora, in order to learn about the long-tail of visual concepts found in the real world. To support and encourage further efforts in this area, I will present the ‘nocaps’ benchmark for novel object captioning. In the second part of the talk, I will describe our recent work on developing agents that follow natural language instructions in reconstructed 3D environments using the R2R dataset for vision-and-language navigation.

[SLIDES]

Speaker Details

Peter Anderson is a Research Scientist in the School of Interactive Computing at Georgia Tech. He completed his PhD in Computer Science at Australian National University in 2018 where he was advised by Prof. Stephen Gould. His research is in computer vision, focusing particularly on the intersection between vision and language including tasks such as image captioning, visual question answering (VQA), and vision-and-language navigation (VLN). He was a member of the team that won the 2017 VQA Challenge. He has published at major computer vision (CVPR, ECCV), natural language processing (ACL, EMNLP), machine learning (NeurIPS) and robotics (ICRA) conferences, and is a co-organizer of workshops and tutorials at CVPR, ECCV, NeurIPS, RSS and ACL.

Series: Microsoft Research Talks