Microsoft Soundscape helps the blind and low vision explore the world in 3D sound with Bing Maps


Imagine being able to navigate through your neighborhood using your hearing alone. For people who are blind or low vision, this is a daily challenge.

Microsoft Soundscape is an application built by the Enable Group in Microsoft Research. It empowers anyone with a visual impairment to experience a mobile voice-based map that offers the independence to explore the world and the ability to choose how to get from place to another. With a stereo headset, a user can traverse their environment using a map delivered in 3D sound. You can hear where landmarks are around you to orient yourself, build a richer awareness of your surroundings, and have the confidence to discover what's around the next corner.

How Microsoft Soundscape enables anyone to hear the world around them

Providing a map to be delivered in 3D through ambient sound, with contextual callouts of what is near a user's location, and orienting a user to an audio beacon as a point of interest is no small feat! So, we asked the team, how does a map in 3D sound work?

According to Melanie Kneisel, a Software Engineer at Microsoft Research, one of the major features in Soundscape is a feature they refer to as automatic callouts. As a person approaches identifiable places and things like businesses, bus stops, and intersections, Soundscape will call out the name of the place or thing from the direction they are located. If you are walking towards a Starbucks that is on your right, you will hear a voice saying "Starbucks" in your right ear to orient you in that direction should you wish to go to Starbucks.

Kneisel continues, "For example, when you approach an intersection the callout is, '36th street goes left, 36th street goes right, 148th continues ahead.' You will hear each of those callouts. When the street goes right, you'll hear it in your right ear. When the street goes left, you'll hear it in your left ear. You start to learn not just that you are approaching an intersection and what is intersecting there, but you learn the geometry of it. You now know that this is not a T intersection, it's a four-way intersection and you know which way each of the roads go."

Woman navigating with Microsoft Soundscape
Source: Microsoft Soundscape

The Soundscape app is not a typical navigation app with turn-by-turn directions. Its key purpose is to empower its user with choice, offering the ability to search their surroundings and take action based on their navigational goal.

When Microsoft Research took on the Soundscape project, the team was faced with a challenge to provide their users with a smarter search experience after receiving feedback that existing map data did not provide the granularity required.

"When this project first came to Microsoft Research, we didn't really have a search tool," says Kneisel. "We were getting feedback that there were issues with the data in Soundscape. We found that a large amount of the feedback was a result of users failing to find certain places within the app while they were trying to accomplish a Soundscape task."

Bing Maps APIs deliver better location search to Soundscape

After evaluating several location search solutions, the Microsoft Research team looked internally and found that Bing Maps had just released two APIs that would meet their requirements — Bing Maps Local Search API and Bing Maps Location Recognition API.

Bing Maps Local Search API is a simple-to-use REST API that allows you to search for businesses by name, category or free text. Soundscape uses Local Search to power the search experience when a user sets an audio beacon or creates a marker. This capability allows a user to search for a specific place by address, name, or category type (e.g., restaurant, shop, etc.).

Bing Maps Location Recognition API offers even more capabilities and returns a comprehensive description of what is near a user's current location. The rich information the API provides includes points of interest with details like business entities (e.g., restaurants, hotels, parks), natural entities (e.g., beaches, islands, lakes), reverse geocoded address of the input location with neighborhood and street intersection information, and even the type of property (e.g., residential, commercial). In the Soundscape app, users can browse a list places nearby their location, mark favorites and set audio beacons for the places they want to visit.

"We chose to use the Location Recognition API because it integrates well with the Bing Maps Local Search API and provides rich category information that we have used to enable category filters on the list of nearby places," Kneisel explains. "Unsurprisingly, accessibility is our team's top priority, so anything that helps Soundscape users to quickly and easily find the information they are looking for is very valuable. For example, the category filters powered by Location Recognition enable users to quickly and easily filter a list of nearby places to view the types of places that they find most interesting."

Better search, better user experience

With the search expertise and powerful services in house with Bing Maps, the Microsoft Research team could leverage the data and search algorithms offered by Bing to deliver a rich experience to Soundscape users without having to build search from scratch.

"We realized that we don't really have a great way to search, nor is our team equipped to build a search engine," says Kneisel. "By using Local Search and Location Recognition to power the experience of finding a place, our team is free to focus on Soundscape-specific tasks, like building the audio beacon experience which helps users to orient or navigate towards a location that they have found using the Bing Maps APIs. "

"Prior to integrating the Bing Maps APIs, users could only search for a place that had previously been cached on their device. This was a very limiting experience because if you hadn't been to a place before, you couldn't search for that place and create a Soundscape marker or set an audio beacon to navigate there," added Kneisel.

Soundscape users are also enjoying the benefits of improved efficiency in the app thanks to the ability of Local Search to handle free text queries, which is imperative for apps using VoiceOver input.

"Local search handles things like typos and it allows people to do category searches and all of the things that just make that experience a lot easier. I really like the typos part because you can save time, especially when you're using something like VoiceOver. I like the affordances it provides," says Kneisel.

Woman navigating with Microsoft Soundscape
Source: Microsoft Soundscape

Lighting up scenarios around mobility and movement is essential for the blind and low vision community whose members primarily walk and rely on public transportation to get around. For Soundscape, including bus stops and the types of entities users will encounter while walking down a street was extremely important, and the Soundscape team is delighted with the progress that Bing Maps is making to continually improve data that better serves the experience of the pedestrian.

With Soundscape already available in the United States, the United Kingdom, Australia and Canada, the team recently announced availability in Sweden with an exciting partnership with SkyView. The SkyView+Soundscape experience is targeted at tourists and visitors to the SkyView globe in Stockholm. By design, Soundscape allows users engage in what is going on around them while being guided by sound cues, which is an innovative and exciting new way to engage with a map. With this as the aspiration driving design decisions behind Soundscape, the Microsoft Research team is looking to extend the technology into areas that will give everyone the opportunity to experience the world lighting up through sound, in new and diverse ways.

Learn more about Soundscape at: www.microsoft.com/en-us/research/product/soundscape/.

For more details about the Bing Maps Platform, go to: www.microsoft.com/maps.