Spoken Language Support for Software Development

Many programmers who suffer from repetitive strain injuries (RSI) and other more severe motor impairments have difficulty staying productive in a work environment that all but requires long hours typing code into a computer. My dissertation work helps to lower these productivity barriers by enabling developers to use speech to reduce their dependence on typing. Many disabled programmers already try to use voice recognition with existing programming environments. However, commercial voice recognition tools do not map well onto programming tasks because they are designed for natural languages. Our research adapts these tools for software development by exploiting the domain-specific nature of programming to constrain the domain of recognition. By applying programming language-based analyses to a verbalized form of authoring, editing and navigating through code, we hope to alleviate many of the common problems of voice recognition for programmers.

Our solution consists of Spoken Java, a semantically isomorphic dialect of Java that is more naturally verbalized by human developers, and a command and control speech grammar designed to enable programmers to find pieces of code and modify them in high- level linguistic ways. Spoken Java is considerably more lexically and syntactically ambiguous than a traditional programming language because verbalized programs contain homophones (words that sound alike but are spelled differently), unpronounceable words, abbreviated words, and inconsistently verbalized punctuation. The major technical challenge in our work is to resolve this new input form’s ambiguities using new algorithms for interacting lexical, syntactic, semantic, and program-specific analyses. The main artifact of our work is a software development system that can understand spoken program authoring, editing and navigation. Our evaluation employs user studies of programmers composing and editing code to learn how well the system performs, and understand how well software developers adapt to voice-based programming. In addition, our software can be used as a platform to learn about more natural modes of interaction than those found in traditional programming environments.

Speaker Details

Andrew Begel is a Computer Science Ph.D. candidate at the University of California, Berkeley. He received his B.S. (1996) and M.Eng. (1997) degrees in Computer Science and Engineering from the Massachusetts Institute of Technology. His research focus lies at the junction of software engineering and human-computer interaction. As part of his dissertation work with Professor Susan L. Graham, Andrew has developed a voice-based programming environment designed to make software development more accessible to programmers suffering from repetitive strain injuries. In collaboration with MIT Professors Mitchel Resnick and Eric Klopfer, Andrew has designed and developed StarLogo, a parallel programming environment that enables high school students and teachers learn about complex systems modeling through simulation.

Andrew Begel
University of California, Berkeley
    • Portrait of Andrew Begel

      Andrew Begel

      Principal Researcher

    • Portrait of Jeff Running

      Jeff Running