“Person, Shoes, Tree. Is the Person Naked?” What People with Vision Impairments Want in Image Descriptions

  • Abigale Stangl
  • Meredith Ringel Morris
  • Danna Gurari

CHI 2020 |

Published by ACM

Access to digital images is important to people who are blind or have low vision (BLV). Many contemporary image description efforts do not take into account this population’s nuanced image description preferences. In this paper, we present a qualitative study that provides insight into 28 BLV people’s experiences with descriptions of digital images from news websites, social networking sites/platforms, eCommerce websites, employment websites, online dating websites/platforms, productivity applications, and e-publications. Our findings reveal how image description preferences vary based on the source where digital images are encountered and the surrounding context. We provide recommendations for the development of next-generation image description technologies inspired by our empirical analysis.

Designing Computer Vision Algorithms to Describe the Visual World to People Who Are Blind or Low Vision

A common goal in computer vision research is to build machines that can replicate the human vision system (for example, detect an object or scene category, describe an object or scene, or locate an object). A natural grand challenge for the artificial intelligence community is to design such technology to assist people who are blind to overcome their real daily visual challenges. In this webinar with Dr. Danna Gurari, Assistant Professor in the School of Information at the University of Texas at Austin, and Dr. Ed Cutrell, Senior Principal Researcher in the Microsoft Research Ability Group, learn how computer vision researchers are working to create vision systems adapted to the needs of those who use them. By creating new dataset challenges, the researchers aim to empower the artificial intelligence community to work on real use cases. To encourage the larger artificial intelligence community to collaborate on developing methods for assistive technology, we introduce the first dataset challenges with data that originates from people who are blind. Our data comes from over 11,000 people in real-world scenarios who were seeking to learn about the physical world around them. More broadly, this dataset serves as a great catalyst for uncovering hard artificial intelligence challenges that must be addressed to create more robust systems across many contexts and scenarios. Together, we’ll explore: Creating tools for people who are blind or have low vision that match their needs and complement their capabilities Key challenges of teaching computers how to automatically describe pictures taken by people who are blind or low vision Several potential solutions to make computers more accurately address the needs of people who are blind or low vision Resource list: 2021 VizWiz Grand Challenge Workshop Image Accessibility (Project page) Ability (Research group) Project Tokyo AI and Accessibility: A Discussion of Ethical Considerations (Publication) "Person, Shoes, Tree. Is the Person Naked?" What People with Vision Impairments Want in Image Descriptions (Publication) Inclusive design for all, or ICT4D and 4U! with Dr. Ed Cutrell (Podcast) AI for Accessibility Grants Ed Cutrell (Researcher profile) Danna Gurari (Researcher profile) *This on-demand webinar features a previously recorded Q&A session and open captioning. This webinar originally aired on March 26, 2020 Explore more Microsoft Research webinars: https://aka.ms/msrwebinars