Microsoft Research Blog

Microsoft Research Blog

The Microsoft Research blog provides in-depth views and perspectives from our researchers, scientists and engineers, plus information about noteworthy events and conferences, scholarships, and fellowships designed for academic and scientific communities.

CodeTalk: Rethinking IDE accessibility

December 11, 2017 | By Suresh Parthasarathy, Senior Research Developer; Gopal Srinivasa, Senior RSDE

From left, Gopal Srinivasa, Manohar Swaminathan, Venkatesh Potluri, Suresh Parthasarathy, and Priyan Vaithilingam.

It is a bright afternoon in the Microsoft Research India lab. Research Fellow, Venkatesh Potluri, sits at his computer, frantically racing against the clock to fix one last bug before the end of the day. The computer blares into his headphones—not music, but a robotic rendition of the code, as Venkatesh uses a program called a screen reader to access applications on his computer.

As his hands glide over the keyboard, the computer reads the current line of code and the contents of the various windows that adorn the integrated development environment (IDE), a state of the art environment he uses for authoring and testing code. Venkatesh steps through the code, listening carefully and concentrating hard to retain in his mind the structures of the code file. He executes the program and after a few seconds the screen reader goes silent. Did the program complete successfully or did it terminate with an exception? Has it hit a break point? Venkatesh is not sure. He groans and decides to add log statements to his program to debug it, frustrated that he would have to spend additional time removing them before submitting his code. Meanwhile, his colleagues invite Venkatesh to join them for a tea break. He refuses, fearing he would lose the context of the file that he painstakingly created in his mind; context that would take precious time to regain—time Venkatesh cannot spare. He types furiously but ends up making a mistake in the code and doesn’t see the squiggly the IDE introduced to highlight the mistake. When he builds his code, he faces a plethora of new errors he has to fix before continuing to fix the original bug. “At this rate, I’m having dinner in the office,” he muses, as he proceeds to find the root cause for the errors on the screen.

IDEs are immensely powerful tools made specifically to improve developer productivity. So why is this one not helping Venkatesh as much? It is mostly because Venkatesh is visually impaired (VI), and many features of IDEs that are a boon to sighted developers are inaccessible to him. He is not alone. In a Stack Overflow survey of programmers, 1 percent of respondents self-identified themselves as being blind, greater than the 0.4 percent of their numbers in the general population. To get a feel for how visually impaired programmers work, check out this demo from a teammate, Saqib Shaikh.

While many IDEs, including Visual Studio and VS Code, are accessible using a screen reader, much of the information from these systems is still conveyed visually. Code is syntax highlighted in bright colors; errors are automatically highlighted using squiggles, and the debugger uses several windows to provide full context of a running program. Performance analysis tools use charts and graphs to highlight performance bottlenecks and architecture analysis tools use graphical models to show the structure of code. These advanced features are generally not accessible to developers with visual impairments, because of which they cannot exploit IDEs to the fullest.

In Venkatesh’s words: “I find the experience of using an IDE with a screen reader incomplete. IDEs provide a lot of relevant information while writing code; A lot of this information such as the current state of the program being debugged, real-time error alerts, and code refactoring suggestions are not announced to screen reader users. As a developer relying on a screen reader, the augmentation IDEs provide is not of high value to me.”

Venkatesh joined the lab’s research in spatial audio for accessibility in early 2017. Suresh Parthasarathy, Senior Research Developer in the lab who interviewed Venkatesh, says, “I was amazed to see how well Venkatesh went about solving the problems I posed and the innovative measures he took to overcome accessibility issues in the IDE. At the same time, I realized that there were many improvements that needed to be made to make IDEs truly beneficial for VI developers.”

Soon, fellow Research Developer Gopal Srinivasa and Research Fellow Priyan Vaithilingam joined Suresh and Venkatesh on a mission, CodeTalk, to enhance the experience of visually impaired developers. As word spread, Saqib Shaikh, an engineer from Microsoft Research Cambridge with VI who had been experimenting with similar ideas, joined forces with them.


“We realized that while a lot of work had gone into making IDEs more accessible, it had fallen short of meeting the unmet and unarticulated needs of visually impaired developers.”  — Gopal Srinivasa


The team realized that for their project to work, they had to start from first principles. In addition to the team’s personal experiences, they drew insights from a survey of visually impaired developers conducted by researcher Manohar Swaminathan with his intern Vidhya. Their efforts culminated in a Visual Studio extension that provided several features to improve the experience of VI developers. Visual Studio was chosen because it is the most popular IDE among developers, supports multiple programming languages, and has a rich API that allowed the team to tap into all its features. The CodeTalk extension and source code are now available on GitHub.

The extension offers the following features:

Glanceability

  • Code summary: Provides a summary of code constructs in the current file in an accessible window. Developers can directly navigate to any code construct from the window.
  • Function list: Provides a navigable, accessible list of functions present in the current file.
  • Get context: Describes the current location of the cursor in the code file by displaying in an accessible window, the hierarchy of code structures (blocks, functions, classes and namespaces) that enclose the current location.

The features described above provide VI developers an experience that is analogous to a sighted developer scrolling through a window and being able to quickly identify required information at a glance.

Navigability

  • Move to context: Moves the cursor to the enclosing code structure with a single keystroke. For instance, a developer can quickly move to the start of a loop or conditional block to make changes without loss of context.
  • Skip comment: Assists developers skip large documentation comments to directly get to the following code structure. Skipping a long comment is harder for a visually impaired developer as they are unable to see the visual highlighting offered by the editor.

Together with the glanceability features, these features help visually impaired developers get around the code faster.

Debugging

  • Automated error information: Automatically provides an auditory warning if a file contains syntax errors. This is an auditory hint equivalent to squigglies shown by the IDE when compile errors are present in a file. Developers can then press a keystroke to access the list of errors. This feature prevents accumulation of compile time errors and mitigates the need for frequent code builds to find such errors.
  • Talk points: Provides auditory cues about the execution of a program while debugging. Developers can instruct the extension to play a tone or evaluate and announce the result of an expression when a specific line of code is reached. The developer can also choose to continue execution or break execution at this point. This mitigates two drawbacks of the current debugging experience–it provides an easy way for VI developers to know if a line of code was executed, and it provides a quick view into the state of the program being debugged. Talk points eliminate the need for debug print statements in the code which are both a nuisance and a possible security risk.

Together, these features make debugging and syntax checking—two critical features of IDEs—far more accessible to VI developers.

After developing CodeTalk, the team conducted a study with VI programmers that indicated it was well received. Real-time error information and talk points were particularly appreciated as significant productivity boosters with one user saying, “I love the idea of breakpoints not breaking!”  The team also began using the extension for their own development, and discovered that the features were useful for sighted users as well.

CodeTalk is one step in a long journey of exploring ways to make IDEs more accessible. Additional work needs to be done to define and meet the needs of visually impaired developers. The team has made source code available on GitHub and invites contributors. The Visual Studio extension can be downloaded here.

Credits: CodeTalk team would like to thank Saqib Shaikh from Microsoft UK (check out Seeing AI) for being a sounding board for all our ideas, and Vyshak Jain for creating the CodeTalk video. We would also like to thank the developers who participated in the survey for their valuable insights and feedback.

Related:

Up Next

Programming languages and software engineering

Summer of MakeCode for kids

Microsoft MakeCode is a web-based learning environment for kids and teens to create with technology. MakeCode takes a unique approach to computing education by combining the magic of making with the power of code as a way to make learning about computers and technology more accessible and interesting to a wider audience. We have created easy MakeCode projects that you can do with your kids at home over the summer using a different MakeCode product each week.

Jacqueline Russell

Principal Program Manager

Programming languages and software engineering

Code in the Classroom with Dr. Peli de Halleux

Episode 12, February 15th, 2018 - If you’ve ever wondered if you could find the perfect combination of computer scientist… and Macgyver, look no further than Dr. Peli de Halleux, principal Research Software Design Engineer at Microsoft Research. A key member of the MSR RiSE team, Peli is part of the MakeCode initiative that brings physical computing to classrooms around the country and around the world. Today, Peli talks about the Maker Movement in K-12 education, the hard work that goes on behind the scenes to deliver a “seamless” user experience for both kids and teachers, and how to get children excited about coding through hands on experience in early computer science education.

Microsoft blog editor

Programming languages and software engineering

MakeCode for Minecraft makes learning to code super fun

  A few years ago, my group in Microsoft’s research organization began to experiment with tools that make it possible for kids to learn how to code in the context of Minecraft, the wildly popular game where players build fantastical virtual worlds out of digital blocks, create and play mini-games within the game, and learn […]

Peli de Halleux

Principal RSDE