Exploring the use of open source Large Language Models in Requirements Engineering

Abstract

In the software development lifecycle, creating Software Requirements Specification (SRS) documentation is an essential yet time-consuming process. The potential of open-source Large Language Models (LLMs) to automatically generate excellent Software Requirement Specification (SRS) documents from transcripts of stakeholder meetings is examined in this thesis. Using a structured prompt and chunked generation technique, the capacity of three open-source LLMs Llama 3.1, Mistral, and Gemma to extract requirements and produce SRS documents that conform to IEEE 29148:2018 guidelines was assessed. Three different project scenarios were used for the evaluation, and each model produced a single complete SRS for each scenario. The texts' consistency, relevance, completeness, format conformance, assumption transparency, and general quality were evaluated by human reviewers. The findings revealed that although each model showed a range of proficiency, Gemma continuously received the top scores for completeness and structural quality, whereas Llama 3.1 excelled in terms of applicability to stakeholder inputs. Mistral performed worse than average in a number of areas. The results suggest that open-source LLMs can be useful for semi-automating requirements engineering when directed by thoughtfully crafted prompts. Full standard compliance and consistent assumption tagging are two areas that still present difficulties. In addition to highlighting important avenues for further research, this work adds to existing efforts to investigate the real-world uses of LLMs in requirements engineering.