GitHub Copilot Developer Experience: A Study Comparing Novice and Experienced Developers' Perspectives

Abstract

With the ever-growing role that AI coding tools play in software development, evaluating the developer experience of using AI-based coding assistants becomes more important than ever before. So far, little attention has been paid to how the developer experience with AI coding assistants may differ among less experienced and more experienced developers. This thesis focuses on GitHub Copilot, a popular AI coding assistant, and investigates the differences in the developer experience among novice and experienced developers and asks whether or not the AI assistant should take into account the developer’s programming experience level. It also explores how the usage of acceleration and exploration modes differ among novice and experienced developers. In the Related Work chapter, existing literature on AI coding assistants and developer experience is reviewed, covering usability, interaction modes, educational impact, and cognitive load. Acceleration and exploration modes are introduced, and the definitions of developer experience are discussed. The Methodology chapter describes the research methods used in the thesis – the online developer survey and developer user study – and justifies their choice. This chapter details online survey distribution, the questions asked, use study participant selection, and the programming tasks given to the participants. The Results chapter presents findings from both the online survey and the user study. Novices used Copilot primarily for debugging and learning, while experienced developers applied it more broadly, however, they were also more critical of context-awareness and security. Exploration mode was more prevalent than acceleration mode among both novices and experienced developers, and experienced developers were more likely to enter acceleration mode. The Discussion chapter interprets the results in relation to the research questions. It concludes that both novices and experienced developers wished to remain in control and were unlikely to delegate all the work to GitHub Copilot. Moreover, the importance of improved onboarding for AI coding assistants is emphasized. A “learning mode” is proposed for AI coding assistants to adapt to developers’ familiarity with tools and frameworks, enhancing onboarding and personalization. The Conclusions chapter presents final reflections, threats to validity (such as small sample sizes) and future research directions (such as different programming languages and frameworks).