Inventing Virtual Reading Teachers and Virtual Speech Therapists

  • Ron Cole and Wayne Ward | Center for Spoken Language Research, University of Colorado Boulder

Advances in computing and interface technologies have made possible the development of a new generation of human computer interfaces that enable natural face to face conversational interaction with virtual humans—3D computer characters that speak, emote and gesture much like human experts in specific task domains. The key challenges in this emerging field involve designing systems that are both engaging and effective; that is, the virtual teacher or therapist is able to engage the clients in immersive experiences that lead to positive learning or clinical outcomes. This requires understanding and modeling well the verbal and nonverbal behaviors of the virtual human during conversational interaction with the client. This talk provides the rationale for and presents demonstrations of two systems that attempt to achieve these goals: one that uses a virtual human to teach children to read and understand text, and one that uses a virtual human to teach adults with Parkinson disease to improve their speech communication abilities.

Speaker Details

Ron Cole is a Professor of Computer Science and director of the Center for Spoken Language Research at the University of Colorado, where he manages research and development efforts that use virtual humans. The most ambitious project, called Foundations to Literacy, teaches young children to read and learn from text through conversational interaction with a virtual tutor. The tutor teaches students the basic skills that underlie reading (e.g., letter names and sounds, phonological awareness, word recognition, spelling), narrates stories to beginning readers, “listens” to students as they read out loud, and assesses comprehension. The program is currently being evaluated in 40 Colorado classrooms for its ability to improve student’s reading skills and comprehension of text. Virtual therapists are also being incorporated into applications to conduct speech therapy with individuals with Parkinson disease, and with individuals with aphasia. In each of these projects, the virtual therapist is designed to emulate the social dynamics—the exchange of verbal and nonverbal signals and cues—between patients and therapists during the course of treatments with proven effectiveness. Clinical trials with virtual speech therapy systems will soon be conducted to asses their effectiveness; successful outcomes of these trials could lead to accessible and inexpensive computer-based treatment systems that improve speech communication skills of millions of individuals with neurological disabilities. Both the Foundations to Literacy program and the virtual speech therapy projects serve as ideal test beds for evaluating research advances in machine perception, human language and character animation technologies that work together to realize virtual humans.Before joining University of Colorado, Professor Cole taught at Oregon Graduate Institute, at Carnegie Mellon University, and at University of Waterloo. Among many accomplishments, Prof. Cole is well known for his work on speech spectrogram reading and its scientific evaluation (with Victor Zue).

Watch Next