VERSE: Bridging Screen Readers and Voice Assistants for Enhanced Eyes-Free Web Search

ASSETS 2019 |

Published by ACM

Best Paper Nominee

People with visual impairments often rely on screen readers when interacting with computer systems. Increasingly, these individuals also make extensive use of voice-based virtual assistants (VAs). We conducted a survey of 53 people who are legally blind to identify the strengths and weaknesses of both technologies, and the unmet opportunities at their intersection. We learned that virtual assistants are convenient and accessible, but lack the ability to deeply engage with content (e.g., read beyond the first few sentences of an article), and the ability to get a quick overview of the landscape (e.g., list alternative search results & suggestions). In contrast, screen readers allow for deep engagement with content (when content is accessible), and provide fine-grained navigation & control, but at the cost of reduced walk-up-and-use convenience. Based on these findings, we implemented VERSE (Voice Exploration, Retrieval, and SEarch), a prototype that extends a VA with screen-reader-inspired capabilities, and allows other devices (e.g., smartwatches) to serve as optional input accelerators. In a usability study with 12 blind screen reader users we found that VERSE meaningfully extended VA functionality. Participants especially valued having access to multiple search results and search verticals.

Publication Downloads

Accessible Information Seeking

August 14, 2020

Slides from ACM CHIIR 2020 closing keynote on Accessible Information Seeking.

Download Data

VERSE: Voice. Exploration. Retrieval. Search.

People with visual impairments are expert users of audio interfaces, including voice-activated virtual assistants and screen readers. Through interviews and surveys of this population, we learned that virtual assistants are convenient and accessible, but lack the ability to deeply engage with content (for example, to read beyond the first sentence of a Wikipedia article), and the ability to present a quick overview of the information landscape (for example, to list other search results and search verticals). In contrast, traditional screen readers are powerful and allow for deeper engagement with content (when content is accessible), but at the cost of increased complexity and decreased walk-up-and-use convenience. Our prototype, VERSE (Voice Exploration, Retrieval, and SEarch), combines the positive aspects of virtual assistants and screen readers to better support free-form, voice-based web search. As with screen readers, VERSE addresses the need to provide shortcuts and accelerators for common actions. Specifically, VERSE allows users to perform gestures on a companion device such as a phone or smart watch. These companion devices are not strictly necessary, but help overcome the long activation phrases that can become tedious when repeated to smart speakers.