Microsoft Research Blog

Microsoft Research Blog

The Microsoft Research blog provides in-depth views and perspectives from our researchers, scientists and engineers, plus information about noteworthy events and conferences, scholarships, and fellowships designed for academic and scientific communities.

Insights on the future of video calling in emergency situations

May 16, 2018 | By Microsoft blog editor

For most of us, a call to emergency services is a rare act, or one thankfully that we’ve never had to take. Anyone having had to make that call often will, in a calmer future moment, reflect on what transpired during it – the quality of the service, the professionalism and empathy of the call takers, and perhaps most importantly how information crucial to the outcome was conveyed. The communication aspect of emergency calls can mean the difference between life and death and 911 call takers are rigorously trained and experienced in guiding callers in order to extract and analyze information and send the needed assistance.

For most of its history, communication in emergency services calls has relied on the ability of the caller to convey information orally. In an age of ubiquitous video calling, this is going to change; over the coming years, emergency calling services in North America will begin to incorporate processes and technologies for video-based calling. The challenge isn’t that the underlying technology isn’t yet ready; it is. What isn’t yet clear is what unique benefits and challenges emergency video calling might bring.

A team of thoughtful researchers at Microsoft Research, Simon Fraser University and the University of Calgary decided to examine and understand the issue of incorporating video in emergency calling services and gleaned significant insights on what might work – and not work – and why. In pursuing these insights, initially from the point of view of the professionals on the receiving end of these calls, they gained a deep respect and appreciation for the almost hidden world of the emergency call takers and what they do while defining the shape of a new research space and the issues involved. The resulting paper, The Benefits and Challenges of Video Calling in Emergency Situations was presented recently at the 2018 ACM CHI Conference on Human Factors in Computing Systems in Montreal. The research and findings will be of keen interest to everyone from emergency planning professionals, to mayors, city planners, federal agencies, technology companies and designers of mobile platforms and, perhaps especially, to lay people.

The idea for examining the ramifications of video calling in emergency situations emerged from conversations between Carman Neustaedter of Simon Fraser University’s School of Interactive Arts and Technology and Kenton O’Hara of Microsoft Research in Cambridge, United Kingdom. Neustaedter, along with his Simon Fraser colleague Brennan Jones, had long been interested in the use of mobile devices for video conferencing in the wild. With a shared interest collaborating on ideas for new uses of technology in health, Neustaedter and O’Hara lighted on the idea of the advent of video calling in emergency situations and what it might mean for people on both ends of the call – those calling for help and, relevant to this initial study, the call takers.

In setting out to understand the daily life and methods of the emergency call takers, the researchers spent a lot of quality time in the call takers’ world, sitting in on real-life emergency calls, conducting contextual interviews at three emergency response call centers in Canada and focusing on the practices of call takers and dispatchers. They came to realize that video calls could provide valuable contextual information about situations and could help overcome call taker challenges having to do issues such as information ambiguity, location, deceit and communication issues (such as poor English-speaking abilities). But they also learned that that video calling in emergencies has the potential to introduce issues involving control, information overload and privacy if systems are not designed well. The findings encourage deeper and expanded research and point to a design space rich with possibilities and potential problems that will need to be addressed.

“We saw a huge need but also a really challenging design space,” said Neustaedter.” “The call takers have a deep appreciation for the delicate balance that they walk – they are very good and have time-tested procedures for extracting the needed information from understandably panicked callers. At the same time, they see the technology that’s going to be coming to their work practices in the next few years and they don’t want that to mess up what works right now. They want their processes to get better but they’re not sure how it’s going to affect them and how it should work.”

“This paper acts as a call to the research community and industry to look at how to address the complexities and challenges of video emergency calling and how to make the most out of the potential opportunities that could emerge from introducing these systems.” – Brennan Jones

Controlling the call

The feedback gleaned by the researchers through their interviews with numerous call takers and dispatchers makes for some of the most fascinating reading in their paper. From the perspective of the call takers, 911 emergency calling is largely about benevolent control, of the caller and the situation. It’s about acquiring information in a specific order and manner that reflects the need for pertinent information while balancing situations in stressful and challenging circumstances.
The ability of call takers to take control of an emergency call is impressive. On oral calls, this is accomplished via spoken communication protocols that call takers are well trained and versed in employing. But the researchers found that video emergency calling will introduce a need for a level of control not seen before in existing literature around video chat or mobile video calling and it suggests that new protocols and sequencing will need to be created to prescribe what information should be shown and in what order over a video call, as well as how call takers should act on that information in order to ensure the best care while limiting liability issues.

Yet control in an emergency video call would be about much more than this; it requires deciding when video and how video should be used. The call takers who participated in the study had numerous concerns when it came to their own autonomy, that is, being able to choose what they would see on video. In a high stress job with frequent emotional pressures associated with empathizing with callers in very distressing circumstances, the desire for autonomy by the call takers is understandable. Despite the potential value for assessing scenes and gaining additional contextual information, video should then be considered secondary in nature for emergency calling, where it would augment audio calls, not replace them.

Essentially, emergency video calls must be considered very differently from conventional video calls; the notion of what is considered video might vary between live one-way video all the way to an audio call that involves images or video clips. Being able to fluidly move between these different visual modalities as part of a video call would be crucial, the researchers concluded.

Camera Work

Control of a 911 video call would also be about the specific camera work needed to capture and manage the information being conveyed. Getting value out of video during a 911 call might rely a lot on the ability of the caller to provide the necessary camera views allowing call takers to understand what they are seeing and to acquire needed information. There is no comparison between these scenarios and the everyday use of Skype to video call with family and friends; the specifics of what needs to be shown in the latter are far less critical. The researchers conclude that this suggests critical design implications to aid callers in performing camera work, perhaps even including a role for AI and automation.
The video option shows promise on a number of levels. Yet the researchers are aware that the presence of video can change the nature of conversations, make them less efficient. Again, the findings suggest a need for a range of visual options in 911 video calls.

Had any of the researchers ever had to call 911? “I have,” said Neustaedter. “I could really empathize with what the call takers in our study were hearing because I’d been in those situations before. It’s one of those things in life that you don’t forget; you have rich memories of the experience and you can imagine how it could change with technology. This memory ability was invaluable in our research involving the callers.”

Next Steps

The researchers recently completed a study that looks at video emergency calling from the point of view of emergency callers and will present their paper, Caller Need and Reactions to 9-1-1 Video Calling for Emergencies at the Designing Interactive Systems Conference in Hong Kong in early June.

Neustaedter spoke of the important roles to be played by multiple entities, from national and local governments, to schools and technology companies on the journey to adoption of beneficent emergency video calling systems. Indeed, because smart phones are corporate products, platform and device companies like Microsoft, Apple and Google will almost certainly be crucial to the effort.

That this research is getting a lot of attention attests to the serious role emergency services play in communities throughout the world and the faith that all of us, regardless of our background or means must place in such services when we least expect it. This importance is reflected in the passion the researchers exude for this project and Microsoft Research is looking forward to an emerging research space accompanied by very interesting stories and thoughtful technologies. As Neustaedter put it eloquently, “We don’t call 911 very often thankfully. But when we do, it’s super important to get it right.”

Up Next

Platform for Situational Intelligence

Artificial intelligence, Human language technologies, Human-computer interaction

Platform for Situated Intelligence: Tools and Framework for Multimodal Interaction Research

Over the last decade, advances in machine learning coupled with the availability of large amounts of data have led to significant progress on long-standing AI challenges. In domains like computer vision, speech recognition, machine translation and image captioning, machines have reached and sometimes even exceeded human performance levels on specific problem sets. However, building end-to-end, […]

Dan Bohus

Senior Researcher

Artificial intelligence, Computer vision, Human-computer interaction, Social sciences

When Psychology Meets Technology with Dr. Daniel McDuff

Episode 17, March 28, 2018 - Dr. McDuff talks about why we need computers to understand us, outlines the pros and cons of designing emotionally sentient agents, explains the technology behind CardioLens, a pair of augmented reality glasses that can take your heartrate by looking at your face, and addresses the challenges of maintaining trust and privacy when we’re surrounded by devices that want to know not just what we’re doing, but how we’re feeling.

Microsoft blog editor

Computer vision, Graphics and multimedia, Human-computer interaction

Typing in the Virtual Office Environment: Examining the Effects of Keyboard and Hand Representation in VR

Much of the hype around Virtual Reality, or VR, has focused on immersive gaming and entertainment, but the technology also has profound implications for enhancing the way we work. Imagine the perfect office environment: multiple large displays, proper lighting, free from distractions and isolated from outside disturbances. VR can be used to generate the ideal virtual office—even from the confines of an airplane seat or a tiny touchdown space.

Eyal Ofek

Senior Researcher