From left, Gopal Srinivasa, Manohar Swaminathan, Venkatesh Potluri, Suresh Parthasarathy, and Priyan Vaithilingam.
It is a bright afternoon in the Microsoft Research India lab (opens in new tab). Research Fellow, Venkatesh Potluri (opens in new tab), sits at his computer, frantically racing against the clock to fix one last bug before the end of the day. The computer blares into his headphones—not music, but a robotic rendition of the code, as Venkatesh uses a program called a screen reader to access applications on his computer.
As his hands glide over the keyboard, the computer reads the current line of code and the contents of the various windows that adorn the integrated development environment (IDE), a state of the art environment he uses for authoring and testing code. Venkatesh steps through the code, listening carefully and concentrating hard to retain in his mind the structures of the code file. He executes the program and after a few seconds the screen reader goes silent. Did the program complete successfully or did it terminate with an exception? Has it hit a break point? Venkatesh is not sure. He groans and decides to add log statements to his program to debug it, frustrated that he would have to spend additional time removing them before submitting his code. Meanwhile, his colleagues invite Venkatesh to join them for a tea break. He refuses, fearing he would lose the context of the file that he painstakingly created in his mind; context that would take precious time to regain—time Venkatesh cannot spare. He types furiously but ends up making a mistake in the code and doesn’t see the squiggly the IDE introduced to highlight the mistake. When he builds his code, he faces a plethora of new errors he has to fix before continuing to fix the original bug. “At this rate, I’m having dinner in the office,” he muses, as he proceeds to find the root cause for the errors on the screen.
Spotlight: Blog post
Research Focus: Week of September 9, 2024
Investigating vulnerabilities in LLMs; A novel total-duration-aware (TDA) duration model for text-to-speech (TTS); Generative expert metric system through iterative prompt priming; Integrity protection in 5G fronthaul networks.
IDEs are immensely powerful tools made specifically to improve developer productivity. So why is this one not helping Venkatesh as much? It is mostly because Venkatesh is visually impaired (VI), and many features of IDEs that are a boon to sighted developers are inaccessible to him. He is not alone. In a Stack Overflow survey of programmers, 1 percent of respondents self-identified themselves as being blind, greater than the 0.4 percent of their numbers in the general population. To get a feel for how visually impaired programmers work, check out this demo (opens in new tab) from a teammate, Saqib Shaikh.
While many IDEs, including Visual Studio and VS Code, are accessible using a screen reader, much of the information from these systems is still conveyed visually. Code is syntax highlighted in bright colors; errors are automatically highlighted using squiggles, and the debugger uses several windows to provide full context of a running program. Performance analysis tools use charts and graphs to highlight performance bottlenecks and architecture analysis tools use graphical models to show the structure of code. These advanced features are generally not accessible to developers with visual impairments, because of which they cannot exploit IDEs to the fullest.
In Venkatesh’s words: “I find the experience of using an IDE with a screen reader incomplete. IDEs provide a lot of relevant information while writing code; A lot of this information such as the current state of the program being debugged, real-time error alerts, and code refactoring suggestions are not announced to screen reader users. As a developer relying on a screen reader, the augmentation IDEs provide is not of high value to me.”
Venkatesh joined the lab’s research in spatial audio for accessibility in early 2017. Suresh Parthasarathy (opens in new tab), Senior Research Developer in the lab who interviewed Venkatesh, says, «I was amazed to see how well Venkatesh went about solving the problems I posed and the innovative measures he took to overcome accessibility issues in the IDE. At the same time, I realized that there were many improvements that needed to be made to make IDEs truly beneficial for VI developers.»
Soon, fellow Research Developer Gopal Srinivasa (opens in new tab) and Research Fellow Priyan Vaithilingam (opens in new tab) joined Suresh and Venkatesh on a mission, CodeTalk (opens in new tab), to enhance the experience of visually impaired developers. As word spread, Saqib Shaikh, an engineer from Microsoft Research Cambridge with VI who had been experimenting with similar ideas, joined forces with them.
“We realized that while a lot of work had gone into making IDEs more accessible, it had fallen short of meeting the unmet and unarticulated needs of visually impaired developers.” — Gopal Srinivasa
The team realized that for their project to work, they had to start from first principles. In addition to the team’s personal experiences, they drew insights from a survey of visually impaired developers conducted by researcher Manohar Swaminathan (opens in new tab) with his intern Vidhya. Their efforts culminated in a Visual Studio extension that provided several features to improve the experience of VI developers. Visual Studio was chosen because it is the most popular IDE among developers, supports multiple programming languages, and has a rich API that allowed the team to tap into all its features. The CodeTalk extension and source code are now available on GitHub (opens in new tab).
The extension offers the following features:
Glanceability
- Code summary: Provides a summary of code constructs in the current file in an accessible window. Developers can directly navigate to any code construct from the window.
- Function list: Provides a navigable, accessible list of functions present in the current file.
- Get context: Describes the current location of the cursor in the code file by displaying in an accessible window, the hierarchy of code structures (blocks, functions, classes and namespaces) that enclose the current location.
The features described above provide VI developers an experience that is analogous to a sighted developer scrolling through a window and being able to quickly identify required information at a glance.
Navigability
- Move to context: Moves the cursor to the enclosing code structure with a single keystroke. For instance, a developer can quickly move to the start of a loop or conditional block to make changes without loss of context.
- Skip comment: Assists developers skip large documentation comments to directly get to the following code structure. Skipping a long comment is harder for a visually impaired developer as they are unable to see the visual highlighting offered by the editor.
Together with the glanceability features, these features help visually impaired developers get around the code faster.
Debugging
- Automated error information: Automatically provides an auditory warning if a file contains syntax errors. This is an auditory hint equivalent to squigglies shown by the IDE when compile errors are present in a file. Developers can then press a keystroke to access the list of errors. This feature prevents accumulation of compile time errors and mitigates the need for frequent code builds to find such errors.
- Talk points: Provides auditory cues about the execution of a program while debugging. Developers can instruct the extension to play a tone or evaluate and announce the result of an expression when a specific line of code is reached. The developer can also choose to continue execution or break execution at this point. This mitigates two drawbacks of the current debugging experience–it provides an easy way for VI developers to know if a line of code was executed, and it provides a quick view into the state of the program being debugged. Talk points eliminate the need for debug print statements in the code which are both a nuisance and a possible security risk.
Together, these features make debugging and syntax checking—two critical features of IDEs—far more accessible to VI developers.
After developing CodeTalk, the team conducted a study with VI programmers that indicated it was well received. Real-time error information and talk points were particularly appreciated as significant productivity boosters with one user saying, “I love the idea of breakpoints not breaking!” The team also began using the extension for their own development, and discovered that the features were useful for sighted users as well.
CodeTalk is one step in a long journey of exploring ways to make IDEs more accessible. Additional work needs to be done to define and meet the needs of visually impaired developers. The team has made source code available on GitHub (opens in new tab) and invites contributors. The Visual Studio extension can be downloaded here (opens in new tab).
Credits: CodeTalk team would like to thank Saqib Shaikh from Microsoft UK (check out Seeing AI (opens in new tab)) for being a sounding board for all our ideas, and Vyshak Jain for creating the CodeTalk video. We would also like to thank the developers who participated in the survey for their valuable insights and feedback.
Related:
Programming languages and software engineering