Customer Story

Microsoft Soundscape is an application built by the Enable Group in Microsoft Research. The Soundscape app is breaking barriers and opening up new possibilities for visually impaired people with voice-based navigation. Anyone can take this app on the go and enjoy the independence that comes with being able to explore the world on their own terms. Using a stereo headset such as AirPods, users can traverse new and old environments guided by a map delivered in 3D sound.

With Microsoft Soundscape, you’ll be able to hear landmarks around you for orientation and navigate with a rich awareness of your surroundings in real time.

Accessible maps for the visually impaired

Soundscape is a pioneer in accessible apps that increase mobility, and it redefines what location data can do. Providing a map to users through ambient sound that includes contextual callouts and 3D sound isn’t easy, so we asked the team about 3D sound and how it affects mapping.

According to Melanie Kneisel, a Software Engineer at Microsoft Research, one of the major features in the Microsoft Soundscape app is a feature they refer to as automatic callouts. As a person approaches identifiable places like businesses, bus stops, and intersections, Soundscape will call out the name of the place from the direction in which they are located. If you are walking towards a Starbucks that is on your right, you will hear a voice saying “Starbucks” in your right ear to orient you in that direction should you wish to go to Starbucks.

The Soundscape app is not a typical navigation app with turn-by-turn directions.

Melanie Kneisel | Software Engineer

Kneisel continues, “For example, when you approach an intersection the callout is, ’36th street goes left, 36th street goes right, 148th continues ahead.’ You will hear each of those callouts. When the street goes right, you’ll hear it in your right ear. When the street goes left, you’ll hear it in your left ear. You start to learn not just that you are approaching an intersection and what is intersecting there, but you learn the geometry of it. You now know that this is not a T intersection, it’s a four-way intersection and you know which way each of the roads go.”

Microsoft Soundscape differentiates itself from the typical navigation apps that provide step-by-step instructions on where and when to take a turn. The Soundscape app aims to place the power of choice in users’ hands, giving them the freedom to define their own routes by providing them with accurate contextual information in real time. Visually impaired users can scan their surroundings to make their own optimal decisions, accessibly and without consulting a map.

When the team took on the Microsoft Soundscape project, they had their work cut out for them. The challenge was overcoming the lack of granularity in existing map data and empowering users with a truly intelligent searching experience — something along the lines of the Autosuggest API.

“When this project first came to Microsoft Research, we didn’t really have a search tool,” says Kneisel. “We were getting feedback that there were issues with the data in Soundscape. We found that a large amount of the feedback was a result of users failing to find certain places within the app while they were trying to accomplish a Soundscape task.”

Creating a location-aware Soundscape app with Bing Maps API

The Microsoft Research team looked at several different candidates to zone in on the perfect location search solution for Microsoft Soundscape. It just so happened that the right answer was an internal solution. Bing Maps’ two new APIs were perfect for their needs — Bing Maps Local Search API and Bing Maps Location Recognition API.

“We chose to use the Location Recognition API because it integrates well with the Bing Maps Local Search API and provides rich category information that we have used to enable category filters on the list of nearby places,” Kneisel explains. “Unsurprisingly, accessibility is our team’s top priority, so anything that helps Soundscape users to quickly and easily find the information they are looking for is very valuable. For example, the category filters powered by Location Recognition enable users to quickly and easily filter a list of nearby places to view the types of places that they find most interesting.”

Better search, better user experience

With experience in providing users with intelligent search solutions and powerful in-house APIs, the team had a head start on developing the Microsoft Soundscape app, avoiding the hassle of having to start from the ground up. Not only were the Bing Maps APIs easy to work with, but they also contributed a range of useful features.

“We realized that we don’t really have a great way to search, nor is our team equipped to build a search engine,” says Kneisel. “By using Local Search and Location Recognition to power the experience of finding a place, our team is free to focus on Soundscape-specific tasks, like building the audio beacon experience which helps users to orient or navigate towards a location that they have found using the Bing Maps APIs. “

“Prior to integrating the Bing Maps APIs, users could only search for a place that had previously been cached on their device. This was a very limiting experience because if you hadn’t been to a place before, you couldn’t search for that place and create a Soundscape marker or set an audio beacon to navigate there,” added Kneisel.

The Local Search API adds further efficiency to the Microsoft Soundscape app with the ability to handle free text queries, an essential quality for apps that rely on VoiceOver input.

“Local search handles things like typos and it allows people to do category searches and all of the things that just make that experience a lot easier. I really like the typos part because you can save time, especially when you’re using something like VoiceOver. I like the affordances it provides,” says Kneisel.

As the visually impaired community relies heavily on public transportation and walking to get around, creating a smooth Soundscape app experience around mobility and movement scenarios was essential for the Microsoft Soundscape team. Soundscape’s contextual information includes nearby bus stops and other useful bits of contextual map data that visually impaired users might find crucial for accessibility.

With Soundscape already available in the United States, the United Kingdom, Australia and Canada, the team announced availability in Sweden with an exciting partnership with SkyView. The SkyView+Soundscape experience is targeted at tourists and visitors to the SkyView globe in Stockholm.

Driven by the aspirations of serving visually impaired communities and making navigation accessible for all, the Microsoft Soundscape team is continually working to evolve and extend this technology into new spaces, illuminating the world through sound.

Learn more about Soundscape at: www.microsoft.com/en-us/research/product/soundscape.

For more details about the Bing Maps Platform, go to: www.microsoft.com/en-us/maps.