{"id":4812,"date":"2015-07-08T09:00:34","date_gmt":"2015-07-08T16:00:34","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/msr_er\/?p=4812"},"modified":"2016-07-20T07:29:07","modified_gmt":"2016-07-20T14:29:07","slug":"recent-progress-on-language-and-vision-observations-from-naacl-and-cvpr-2015","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/recent-progress-on-language-and-vision-observations-from-naacl-and-cvpr-2015\/","title":{"rendered":"Recent progress on language and vision: Observations from NAACL and CVPR 2015"},"content":{"rendered":"

I recently had the opportunity to attend two interesting conferences, NAACL (North America Association of Computational Linguistics) and CVPR (Computer Vision and Pattern Recognition). They are top conferences in the fields of natural language processing (NLP) and computer vision, respectively, and traditionally, their audiences are quite different. Along with all the exciting advances, I noticed an interesting coincidence: This year both conferences included a session on language and vision. However, perhaps this is not<\/em> a coincidence. The merging of language and vision is one of the most active areas in NLP and computer vision research, and the community has made significant progress in this area over the past year, largely driven by recent breakthroughs in deep learning technologies and their applications in vision and language.<\/p>\n

When I was at CVPR, I heard one question repeated several times: Why should vision researchers care about language processing? In computer vision, researchers recently built very deep convolutional neural networks (opens in new tab)<\/span><\/a> (CNN), achieved an impressively low error rate in large-scale image classification tasks (opens in new tab)<\/span><\/a>, and even reached human-level image classification accuracy (opens in new tab)<\/span><\/a> early this year. These accomplishments were achieved while requiring little understanding about language processing.<\/p>\n

In order to teach a computer to predict the category of a given image, one way is for researchers to first annotate each image in a training set with a category label (called \u201csupervision\u201d) from a predefined list of 1000 categories. Through trial-and-error training, the computer learns how to classify an image.<\/p>\n

However, it\u2019s a different situation when we want computers to understand<\/em> complex scenes. In such cases, it is usually not possible to define simply by category all fine-grained, subtle differences among these scenes. Instead, the best supervision is full descriptions in natural language.<\/p>\n

\"frisbeephoto-1024x377\"<\/a><\/p>\n

An example from the MS COCO dataset. (Microsoft Research)<\/em><\/p>\n

Although the photos could be quite diverse, each description, annotated by a human, focuses on salient information in the image, delivers a coherent story with clear and consistent semantic meaning, and reflects certain common knowledge. These descriptions provide very rich information about the meaning of the picture, from a human point of view, and could serve as the supervision to train the computer to understand the image as a human does.<\/p>\n

Another problem is how to test whether the computer understands the image. One way is to give the computer an image and ask the computer to generate a descriptive caption about it. We can discover how well the computer understands the image by looking at the generated caption through a Turing Test, in which we weigh the quality of the computer\u2019s output against the output generated by humans for the same photograph.<\/p>\n

For both of these training and evaluation problems, it is necessary for the computer to understand language. As a result, language processing is of particular importance for vision researchers for building strong artificial intelligence in vision.<\/p>\n

In order to facilitate the research of image understanding, Microsoft sponsored the creation of the COCO (Common Objects in Context) dataset (opens in new tab)<\/span><\/a>, the largest image captioning dataset available to the public. The availability of the data was in part the motivation that led some groups from academic and industry institutes to work on solving the auto image captioning problem (opens in new tab)<\/span><\/a> in the last year.<\/p>\n

In order to have a rigorous comparison of different approaches and speed up the research for the community, the MS COCO committee (opens in new tab)<\/span><\/a> organized an image captioning challenge (opens in new tab)<\/span><\/a> at CVPR2015. There were a total of fifteen entries, and the top teams were invited to present their systems at the LSUN workshop (opens in new tab)<\/span><\/a> at CVPR.<\/p>\n

Microsoft tied Google (opens in new tab)<\/span><\/a> for first prize at the competition. The MSR entry had slightly more outputs passing the Turing Test compared to the Google entry, though MSR generated slightly fewer captions that were better or equal to those generated by human annotators.<\/p>\n

\"google-system\"<\/a><\/p>\n

Google\u2019s system uses a two-stage approach for direct caption generation. (Google)<\/em><\/p>\n

Despite the tie, Microsoft and Google have quite different systems. Google uses a CNN to generate a whole-image feature vector, then feed it into a recurrent neural network (RNN) based language model to generate the caption directly.<\/p>\n

In contrast, Microsoft\u2019s system takes a three-stage approach. \u00a0First, we tuned a CNN to detect multiple objects in the input image\u2014not only nouns, but also verbs and adjectives (in this example, woman, crowd, cat, camera, holding, purple<\/em>; note that this stage might introduce detection noise, such as cat<\/em>).<\/p>\n

\"MS-system\"<\/a>Microsoft\u2019s system takes a three-stage approach. (Microsoft Research)<\/em><\/p>\n

Second, we proposed an image-conditioned maximum entropy language model (LM) to generate 500 caption candidates that covered the detected words (a purple camera with a woman, a woman holding a camera in a crowd, a woman holding a cat.<\/em>) The LM injects words such as a<\/em> and with<\/em> to make the caption read fluently.<\/p>\n

Third, we developed a deep multimodal similarity model (DMSM) that re-ranks and selects the optimal caption that captures the overall semantic content of the image. (Note the noise in word detection and LM stages: the DMSM will pick the semantically correct caption a woman holding a camera in a crowd,<\/em> and drop others that don\u2019t match the global semantics of the image such as a purple camera with a woman<\/em> and a woman holding a cat.<\/em>)<\/p>\n

Compared to the CNN->RNN approach Google and others adopted, our system can trace back and locate the region in the image for key words in the caption that the system generated, thanks to the explicit word detection. So it can provide grounded evidences for the generated caption and add interpretability to the outcome.<\/p>\n

The results indicate that the current state-of-the-art system is about halfway to passing the whole Turing Test\u2014that is, the MSR system scored 32.2% while a human scored 67.5%. (Note that even humans cannot pass with 100% since people disagree on the best way to describe an image. This also reflects the difficulty and ambiguity in the task of computer understanding of an image.)<\/p>\n

Continuing in this line of investigation, researchers are already exploring advanced algorithms that translate videos to natural language (opens in new tab)<\/span><\/a> and answer questions related to the content of an image (opens in new tab)<\/span><\/a>. In the future, by connecting speech, language, vision, and knowledge bases, we are looking forward to building a universal intellectual system that would blur the boundary between the machine and the human, with ubiquitous and invisible computational intelligence.<\/p>\n

\"xiaodong\"<\/a><\/p>\n

Xiaodong He<\/strong> is a researcher in the Deep Learning Technology Center (opens in new tab)<\/span><\/a>\u00a0of\u00a0Microsoft Research (opens in new tab)<\/span><\/a> in\u00a0Redmond, Washington. He is also an\u00a0Affiliate Professor (opens in new tab)<\/span><\/a>\u00a0in the Department of Electrical Engineering at the\u00a0University of Washington (opens in new tab)<\/span><\/a>\u00a0(Seattle)\u00a0serving in the PhD reading committee (opens in new tab)<\/span><\/a>. His\u00a0research interests include\u00a0deep learning, speech, natural language, vision, information retrieval, and knowledge management.\u00a0He and\u00a0his colleagues developed the\u00a0MSR-NRC-SRI entry (opens in new tab)<\/span><\/a>\u00a0and the\u00a0MSR entry (opens in new tab)<\/span><\/a>\u00a0which\u00a0were\u00a0No. 1\u00a0 (opens in new tab)<\/span><\/a>in the\u00a02008 NIST MT Evaluation (opens in new tab)<\/span><\/a>\u00a0and the\u00a02011 IWSLT Evaluation (opens in new tab)<\/span><\/a>, Chinese-English, respectively. He and colleagues also developed the\u00a0MSR image captioning system\u00a0 (opens in new tab)<\/span><\/a>that won the\u00a01st Prize (opens in new tab)<\/span><\/a>\u00a0in the\u00a0MS COCO Captioning Challenge 2015 (opens in new tab)<\/span><\/a>. He has published\u00a0in\u00a0Proc. IEEE, IEEE TASLP, IEEE SPM, ICASSP, ACL, EMNLP, NAACL, CVPR, SIGIR, WWW,\u00a0CIKM, ICLR.<\/p>\n

His current research focus is on\u00a0deep learning\u00a0for semantics and applications to text, vision, information retrieval, and knowledge graph (opens in new tab)<\/span><\/a>. Relevant\u00a0studies are summarized in\u00a0the recent\u00a0tutorial at\u00a0CIKM 2014 (opens in new tab)<\/span><\/a>. More details can be found at the\u00a0DSSM\u00a0site (opens in new tab)<\/span><\/a>.<\/p>\n

He has held editorial positions on several IEEE Journals and has served on the organizing committee\/program committee of major speech and language processing conferences. He is a senior member of IEEE and a member of ACL.<\/p>\n

He received a BS degree from\u00a0Tsinghua University (opens in new tab)<\/span><\/a>\u00a0(Beijing) in 1996, MS degree from\u00a0Chinese Academy of Sciences (opens in new tab)<\/span><\/a>\u00a0(Beijing) in 1999,\u00a0and a PhD degree from the\u00a0University of Missouri, Columbia (opens in new tab)<\/span><\/a>, in 2003.<\/p>\n

For more computer science research news, visit ResearchNews.com (opens in new tab)<\/span><\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

I recently had the opportunity to attend two interesting conferences, NAACL (North America Association of Computational Linguistics) and CVPR (Computer Vision and Pattern Recognition). They are top conferences in the fields of natural language processing (NLP) and computer vision, respectively, and traditionally, their audiences are quite different. Along with all the exciting advances, I noticed […]<\/p>\n","protected":false},"author":32627,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194471,194455,194456,194459],"tags":[195101,195138,186925,195882,193504,186936,196682],"research-area":[],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-4812","post","type-post","status-publish","format-standard","hentry","category-computer-vision","category-machine-learning","category-natural-language-processing","category-researchnews","tag-common-objects-in-context-coco-dataset","tag-computer-vision-and-pattern-recognition-cvpr","tag-deep-learning","tag-image-captioning-challenge","tag-microsoft-research","tag-natural-language-processing","tag-north-america-association-of-computational-linguistics-naacl","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"July 8, 2015","formattedExcerpt":"I recently had the opportunity to attend two interesting conferences, NAACL (North America Association of Computational Linguistics) and CVPR (Computer Vision and Pattern Recognition). They are top conferences in the fields of natural language processing (NLP) and computer vision, respectively, and traditionally, their audiences are…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4812","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/32627"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=4812"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4812\/revisions"}],"predecessor-version":[{"id":260739,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/4812\/revisions\/260739"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=4812"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=4812"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=4812"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=4812"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=4812"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=4812"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=4812"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=4812"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=4812"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=4812"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=4812"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}