Towards More Effective AI-assisted Programming: A Systematic Design Exploration to Improve Visual Studio IntelliCode’s User Experience

AI-driven code editor extensions such as Visual Studio IntelliCode and Github CoPilot have become extremely popular. These tools recommend inserting chunks of code, with the lines to be inserted presented inline at the current cursor location as gray text. In contrast to their popularity, other AI-driven code recommendation tools that suggest code changes (as opposed to code completions) have remained woefully underused. We conducted lab studies at Microsoft to understand this disparity and found one major cause: discoverability. Code change suggestions are hard to surface through bold, inline interfaces and hence, developers often do not even notice them. Towards a systematic understanding of code change interfaces, we performed a thorough design exploration for various categories of code changes: additive single-line changes, single-line changes, and multi-line changes. Overall, we explored 19 designs through a series of 7 laboratory studies involving 61 programmers and distilled our findings into a set of 5 design principles. To validate our results, we built and deployed a new version of IntelliCode with two of our new inline interfaces in Microsoft Visual Studio 2022 and found that they lead to a significant increase in usage of the corresponding tools.