Choosing a tool for improved software test management

Abstract

The purpose of this study was to investigate testing management activities of the case organization, find the most important features and criteria for the evaluation of different test management tool alternatives, and finally to assign merit for different alternatives by evaluating different tools so that the organization may select the most appropriate tool for them. The organization had two options: to acquire a commercial tool from an external vendor or to develop one themselves. Based on these objectives four research questions were assigned, for which giving answers would benefit both academics and practitioners.

The study is done in a manner that follows elements of action research and a single case study. It is a multi-method research combining several data collection methods: secondary data, semi-structured interviews, and observations and field notes. Test management is not a well-established practice in the academic literature, but it has become increasingly interesting subject. Software organizations aim to improve the efficiency and effectiveness of their testing activities, as it is very expensive and important part of the quality assurance of a software product. One way to do this is by implementing dedicated tool support around testing. There are various implementations of these test management tools, but they have in common that they facilitate performing test management activities with improved test asset management. The literature identifies certain practices when choosing software systems, but in test management tool context, this is rather a unique study in this regard.

It was discovered in the study that the case organization has various difficulties in test management activities mainly because of difficulties in test asset management. The most important features for the organization were the ability to plan, monitor, and log information within the tool, while the most important criteria included these functional and other non-functional criteria: usability, extensibility, and customizability. The evaluations showed that the commercial tools do fulfill functional criteria to some extent, but they have various limitations that team would need to cope with. Towards the end, an internally developed tool became increasingly beneficial option and it became the recommended alternative. The most important contributions based on the results of this study are both for practical and academic usages. The project and the organization where this study was done benefitted from the results as they gained information about their current processes and needs. The academic community gained a thoroughly documented case including difficulties in a software organization regarding test management, and an insight what kind of criteria are the most important when deciding which alternative is picked for test management solution.