Microsoft Research

Mobile Device Uses Images to Help Children with Autism Interact

Autism spectrum disorder, which affects more than 500,000 U.S. children and millions more worldwide, can make verbal communication a challenge. With support from Microsoft External Research, two U.S. academic researchers are developing mobile device software that can enable individuals with autism to communicate more effectively.

The ability to communicate is essential in helping humans form relationships, learn and participate in society. For many people with autism-a developmental disorder characterized by impaired verbal skills-communicating needs, thoughts and emotions in words can often be intensely frustrating.

Because people with this disorder tend to process information visually, educators and therapists have adopted various approaches to teaching communication skills to students with autism through the use of picture cards. One methodology is the Picture Exchange Communication System (PECS), in which people learn to form simple sentences by arranging images on an adhesive strip-such as pairing the symbol for "I want" with a picture of a cereal box or an apple to choose their breakfast. The cards also help parents, teachers and others to ask questions or give instructions to someone with autism.

Seeing the potential to make PECS-style communication methods even more convenient and interactive, university researchers Gondy Leroy and Gianluca De Leo are developing new software that presents the images on a handheld computing device such as a smartphone. This allows a user to move pictures around to form sentences simply by touching the device screen. The software also tracks which pictures are being used and how often, what sentences the user creates, and other data to help teachers and therapists assess learning progress.

Picture-based communication can be a real breakthrough for children with autism as well as their families and teachers. People are thrilled to use these images and tools on a low-cost, portable electronic device. Gondy Leroy, associate professor of information systems and technology, Claremont Graduate University

Initial trials of the application involving children with moderate to severe autism have yielded encouraging feedback about its advantages over traditional laminated paper cards. "Picture-based communication can be a real breakthrough for children with autism as well as their families and teachers," says Leroy, an associate professor of information systems and technology at Claremont Graduate University in Claremont, California. "People are thrilled to use these images and tools on a low-cost, portable electronic device."

Autism spectrum disorder affects more than 500,000 children in the United States, according to the U.S. Centers for Disease Control and Prevention, and millions more around the world. Experts estimate that it is growing at a rate of between 10 and 17 percent a year. Its symptoms-which vary widely in severity among different people on the autism spectrum-include developmental disabilities, withdrawal, lack of social behavior, and language and attention deficits. While tools such as PECS have been successful in helping people with autism to communicate and even develop greater spontaneous verbal language in many cases, Leroy says electronic devices and software have the potential to build on this approach.

Although Leroy's research background is in information retrieval and text mining, she became interested in developing technology for picture-based communication three years ago after seeing a video about PECS on the Web. "My first thought was, 'Why not put it on a handheld computing device?'" she says. "I asked some clinicians and professors at Claremont if doing that would be useful, and they got really excited."

She learned that users of picture-based communication methods often struggle with managing a three-ring binder full of paper cards that are easily misplaced, damaged or forgotten at home. "One special-education instructor told me, 'I can't stand the smell of laminating paper anymore because I have to make so many pictures!'" says Leroy. Using the cards in public also can feel embarrassing, especially for older children and adults.

Microsoft External Research provided financial and software support to help Leroy and De Leo, an assistant professor in the Virginia Modeling, Analysis and Simulation Center at Old Dominion University, develop their electronic communication device. After meeting with teachers, parents and therapists in several communities to better understand their needs, the researchers completed a prototype of the software in 2008. Leroy and De Leo are each working with small groups of educators and parents of autistic children to gauge the device's effectiveness.

Available as a free download from the project Web site, the software works on any smartphone or personal digital assistant (PDA) device running the Microsoft Windows Mobile operating system. The Web site includes a downloadable repository of more than 400 digitized picture cards in categories such as School, Home, People, Emotions and Food. Parents and teachers can also upload their own photos to create new cards and share them with others.

"We wanted to design a tool that works as closely as possible to how children with autism already use PECS-type cards, so they could start to form messages with very little training," says De Leo. He adds that the application's ability to gather statistics about how children are using these devices and consolidate the data-by individual user or in anonymized reports involving larger groups-will help educators, therapists and autism researchers expand their knowledge of the disorder and how to treat it.

De Leo and Leroy intend to make the software tool available more broadly on the Web and promote it among regional and national autism associations. Plans for enhancing the software include linking it to a calendar so autistic children can view their daily routine in pictures, enabling the handheld device to play short video clips, and adding sound messages to help children associate pictures with words.

Stephen Vaughn, who oversees special-education programs as area director of Desert/Mountain Operations for the San Bernardino County Superintendent of Schools in California, agrees that the new device could help students with autism become more engaged in learning and help teachers provide more effective instruction. "If these kids can communicate their wants and needs more clearly, it will reduce their frustration and open the door for them to interact in a much wider sphere of social life," says Vaughn.

"And because it's on a cool little electronic device, they love it," he adds. "Lots of people today have a smartphone or a PDA, so hopefully children with autism and their families can feel more comfortable using this tool."

(c) 2009 Microsoft Corporation. All rights reserved. This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft and Windows Mobile are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

 

Project Principals:

Gondy Leroy, Ph.D., associate professor, Information Systems and Technology Department, Claremont Graduate University

Gianluca De Leo, Ph.D., assistant professor, Virginia Modeling, Analysis and Simulation Center, Old Dominion University

Web Site:

http://www.communicationautism.org