{"id":646776,"date":"2020-03-31T10:25:51","date_gmt":"2020-03-31T17:25:51","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=646776"},"modified":"2020-04-23T15:57:28","modified_gmt":"2020-04-23T22:57:28","slug":"alt-text-that-informs-meeting-the-needs-of-people-who-are-blind-or-low-vision","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/alt-text-that-informs-meeting-the-needs-of-people-who-are-blind-or-low-vision\/","title":{"rendered":"Alt text that informs: Meeting the needs of people who are blind or low vision"},"content":{"rendered":"
\"Screenshots

In a study about the kinds of details people who are blind or have low vision want when encountering digital images, researchers used several different types of sources as prompts. The above are examples of images participants encountered when browsing some of those sources. Participants reported desiring image descriptions that varied in detail depending on the context in which an image appeared.<\/p><\/div>\n

Image descriptions are vital to making digital content fully accessible to people who are blind or have low vision. However, as past research has shown, content authors often leave out these descriptions\u2014also known as \u201calternative text\u201d or \u201calt text\u201d\u2014for images on the web<\/a> and on social media<\/a>, making AI-based vision-to-language services that automatically generate image descriptions particularly important. To learn more about the design requirements for AI systems to create useful descriptions, our research team conducted interviews with 28 people who are blind or have low vision about the types of details they want when encountering digital images and identified ways in which those wants vary depending on where an image is encountered.<\/p>\n

Our findings are presented in the paper \u201c’Person, Shoes, Tree. Is the Person Naked?’ What People with Vision Impairments Want in Image Descriptions,”<\/a>\u00a0which was accepted at the ACM CHI Conference on Human Factors in Computing Systems (CHI 2020) (opens in new tab)<\/span><\/a>. The work is part of the Microsoft Ability Initiative<\/a>, a two-year collaboration launched last year between Microsoft researchers in accessibility and computer vision and faculty and students in AI and human-computer interaction at The University of Texas at Austin. Funded by Microsoft Research and AI for Accessibility Program<\/a>, the initiative aims to increase the utility and usability of automatically generated image descriptions for people who are blind, a population that relies on screen reader technology to access computing devices. Screen readers present textual content as audio or Braille output and can only present digital images if they have an accompanying alt text description.<\/p>\n

The paper\u2019s title stems from one participant\u2019s observation regarding the inadequacy of current AI-based image descriptions. In the tag-based approach used by Facebook, for example, images are labeled with statements such as \u201cThis image may contain: a person, shoes, a tree.\u201d In describing their encounters with such labels, the participant wondered in a tongue-in-cheek way whether the person in the image was really wearing only shoes, highlighting the lack of critical details in today\u2019s AI-generated descriptions.<\/p>\n

\u201cIn this work, our first priority was to learn from people who are blind or have low vision about their daily experiences with image descriptions,\u201d said our lead author on the paper, postdoctoral researcher Abigale Stangl from UT Austin (opens in new tab)<\/span><\/a>. \u201cThe AI for Accessibility program enabled us to take the time to hear from teachers, lawyers, music producers, students, politicians, job seekers, and others and then communicate our empirical findings in a way that can improve computer vision algorithms.\u201d<\/p>\n

Same image, different details<\/h3>\n

While earlier studies have identified that the usefulness of image descriptions is dependent on the digital context in which the image appears, our study gives insight into how image description wants vary based on context across seven different sources: news websites, social networking sites\/platforms, e-commerce websites, employer\/employment websites, online dating websites\/platforms, productivity applications, and e-publications.<\/p>\n

\u201cThough it is well understood that image descriptions are important to convey the purpose of an image, this research showed us that people who are blind or have low vision want image descriptions that are responsive to where they encounter the image,\u201d said Stangl. \u201cIn other words, people want different content for the same image depending on where they find it.\u201d<\/p>\n

For example, if a photo of a person appeared in a news story, people might want a description that includes details about the setting of the image to give a sense of place. But if a photo of a person appeared on a social media or dating website, people might want increased details about that person\u2019s appearance, including some details that may be subjective and\/or sensitive, such as race, perceived gender, and attractiveness. One participant mentioned that knowing the race and gender of people in photos of board members on an employer\/employment website might help them understand whether the company values a diverse workplace. These latter examples illustrate practical and ethical challenges for emerging AI systems, such as whether AI systems can\u2014or should\u2014be trained to provide subjective judgments or information about sensitive demographic attributes.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/td>\nNews<\/strong><\/td>\nSocial Networking<\/strong><\/td>\neCommerce<\/strong><\/td>\nEmployment<\/strong><\/td>\nDating<\/strong><\/td>\nProductivity<\/strong><\/td>\nE-Publication<\/strong><\/td>\n<\/tr>\n
Event\/Scene<\/strong><\/td>\n<\/tr>\n
People Present<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Text<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Activity<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Interaction<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Landmarks<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Building Features<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Weather<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Lighting<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
People<\/strong><\/td>\n<\/tr>\n
Text<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Salient Objects<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Activity<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Gender<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Race\/Diversity<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Name of Person<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Celebrity Name<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/tr>\n
Expression<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Attire\/Clean<\/td>\n<\/td>\nx<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Body Shape\/Size<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Pets<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Hair Color<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Hair Style<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Eye Color<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Unique Physical<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Tattoos<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Object<\/strong><\/td>\n<\/tr>\n
Text<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Name<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/tr>\n
Form<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Fit<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Color<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Overall Style<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Material<\/td>\nx<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Logos\/Symbols<\/td>\n<\/td>\nx<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Damage<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Unique Features<\/td>\n<\/td>\n<\/td>\nx<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The above table is a cross-source analysis of participants\u2019 description preferences, indicated with an x<\/em>, for seven source types. The preferences are grouped by common focuses of image composition\u2014event\/scene, people, and objects\u2014and the types of preferences were included if at least one participant indicated them as important to include. <\/sub><\/p>\n

As part of our research, we created a tabular reference guide identifying the types of details people find useful across sources. These findings can inform the design of future AI-based captioning tools. For instance, the details of interest to our participants suggest new categories of metadata that should be produced by crowd workers to feed into improved machine learning models. Our work also highlights the opportunity for creating custom vision-to-language models that are dependent on the context in which an image appears.<\/p>\n

Accelerating advancement<\/h3>\n

The Microsoft Ability Initiative with UT Austin is part of the Microsoft Research Ability team\u2019s<\/a> ongoing focus on image accessibility<\/a>. In addition to this research, we\u2019re jointly hosting an Image Captioning Data Challenge (opens in new tab)<\/span><\/a> and VizWiz Grand Challenge Workshop (opens in new tab)<\/span><\/a> as part of the International Conference on Computer Vision and Pattern Recognition (CVPR 2020) (opens in new tab)<\/span><\/a>, which is scheduled to take place in Seattle in June.<\/p>\n

\u201cWe are excited that this event will promote greater interaction between the diverse groups of researchers and practitioners working on image description technologies,\u201d said co-author and UT Austin professor Danna Gurari (opens in new tab)<\/span><\/a>. \u201cUltimately, we expect this work will accelerate the conversion of cutting-edge research into market products that better empower people who are blind or have low vision to independently address their daily visual challenges.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Image descriptions are vital to making digital content fully accessible to people who are blind or have low vision. However, as past research has shown, content authors often leave out these descriptions\u2014also known as \u201calternative text\u201d or \u201calt text\u201d\u2014for images on the web and on social media, making AI-based vision-to-language services that automatically generate image […]<\/p>\n","protected":false},"author":38838,"featured_media":646953,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Meredith Ringel Morris","user_id":"32884"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-646776","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[283244],"related-projects":[638784],"related-events":[641571],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"\"Screenshots","byline":"Meredith Ringel Morris","formattedDate":"March 31, 2020","formattedExcerpt":"Image descriptions are vital to making digital content fully accessible to people who are blind or have low vision. However, as past research has shown, content authors often leave out these descriptions\u2014also known as \u201calternative text\u201d or \u201calt text\u201d\u2014for images on the web and on…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/646776","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=646776"}],"version-history":[{"count":24,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/646776\/revisions"}],"predecessor-version":[{"id":646950,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/646776\/revisions\/646950"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/646953"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=646776"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=646776"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=646776"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=646776"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=646776"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=646776"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=646776"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=646776"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=646776"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=646776"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=646776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}