Online Detection of Change Points in Software Regression Test Case Execution Times

Abstract

In a world where the role of software has become ever more prominent in our daily lives, software testing has at the same time become an increasingly important practice. Although software testing typically involves verifying that a software application produces expected outputs to given inputs, it is often reasonable to test nonfunctional aspects of an application, such as verifying that the performance meets a certain criteria. In addition to the current performance of an application, a potentially useful piece of information is to detect when the performance has changed abruptly at some time point. Such information is particularly useful in the context of regression testing, where an existing functionality of an application is verified to continue working as expected after making changes to the application.

This study presents an online approach for detecting abrupt changes, or often referred to as change points, in the performance of an application. The execution times of automated regression test cases are utilized as an indicator of the performance of an application feature that is being tested in the test case. The goal of online detection in this study is to detect change points with high accuracy and minimal delay after the full suite consisting of multiple test cases has been executed. A sample traversal algorithm that utilizes an online sliding window consisting of two smaller consecutive sub-windows of samples is presented. For a single time step of the algorithm, various change point detection methods are applied for calculating test statistics that measure the dissimilarity of the samples from the two sub-windows. The test statistics are then compared against a threshold that determines the decision of a change point. In addition to test statistic -based change point detection methods, a Bayesian approach that updates the change point probabilities sequentially for each incoming sample is also presented.

The best change point detection performance in terms of the evaluation metrics presented in this study is obtained by the two-sample Kolmogorov-Smirnov test, which calculates a test statistic from the samples in an online sliding window. A real world simulation of change point detection is presented, where the detection performance using unseen samples from a testing set is evaluated with threshold levels for different false alarm rates obtained using samples from a separate training set. The implementation of a change point detection system as part of an existing test automation cloud application named Qentinel Pace is also considered. The end result is a potential implementation architecture of a change point detection system, which utilizes modern cloud native concepts and integrates as part of the application using a microservices architecture.