Developing a Measurement Framework for AI Tool Productivity

Abstract

Developer productivity has been measured for decades using different methodologies and metrics. The topic is especially important due to the recent surge of Generative AI (GenAI) in the field of Software Development. At first glance, it may seem obvious, that GenAI boosts Developer Productivity. While there are sources backing up this claim, there are other sources showing, that usage of GenAI tools may even decrease productivity. Some sources say, that while AI can speed up areas such as programming, it causes bottlenecks in later stages of software development, for example in review or testing stages. Instead of trying to prove, whether AI affects productivity positively or negatively, this thesis investigates how the productivity may or could be measured.

This thesis introduces a framework for measuring AI’s impact on developer productivity. The framework includes existing productivity metrics and frameworks such as DORA, and SPACE. These metrics and frameworks are used to calculate Composite Productivity Score, which can be considered a squeezed metric for productivity. This squeezed metric can then be used with AI-tool usage related metrics utilizing both visual and statistical methods. The framework is implemented as Proof of Concept (PoC), and it is demonstrated using synthetically generated data.

The thesis finds that developer productivity is multifaceted concept, and measuring it requires considering multiple dimensions, such as satisfaction, performance, and activity. Furthermore, since one metric may be more important than another one, it is important for the productivity framework to allow setting different weight for different metrics.

Based on examples with synthetic data, the thesis recommends measuring amount of AI rework, to find out if it correlates with productivity negatively in real-life scenarios as well. The thesis also suggests that measuring delayed impact of AI-usage is important. One example of delayed impact in the context of AI is short term productivity increase, followed by productivity decrease later. The productivity decrease may be explained by overenthusiastic AI-tool usage which leads to high amount of AI generated code.

Since the framework is developed as PoC, the thesis discusses further development opportunities and possible risks related to it. Examples of these development opportunities are integrating it to data sources such as GitHub, setting system-recommended metrics and weights for productivity, and deriving confidence score for the productivity calculations. Finally, the thesis briefly mentions similar tools and frameworks such as Faros AI and LinearB.