A Multi-agent LLM System for Automated Requirements Analysis: A Study on User Story Generation and Prioritization

Abstract

Manually specifying and prioritizing user stories in software projects is time-consuming and prone to inconsistency. This paper investigates whether Large Language Models (LLMs) can support these activities through a role-based multi-agent system. The proposed system uses four models: GPT-3.5 Turbo, GPT-4o, LLaMA 3.3, and Mistral-Nemo, to generate user stories from a project description and prioritize them using prompts simulating stakeholder roles. Relevance is evaluated using semantic similarity to the project description, and prioritization consistency is assessed using Kendall’s Tau distance against expert rankings. Results indicate that all models generate functionally relevant requirements with high semantic similarity, although clarity and conciseness vary. In prioritization, the models show moderate alignment with expert rankings, particularly for mid- and low-priority items, while variability across runs remains a challenge.