Evaluating Transformer and LLM Constraints for Traceability Link Recovery

Abstract

Modern software evolves rapidly, and the rationale behind changes often fades as issues are closed and commits accumulate. This makes it difficult for developers to determine which commit implemented a given issue or requirement, or which issue explains a particular code change. Requirements traceability addresses this problem by linking related development artifacts, but such links are rarely maintained consistently in real GitHub repositories. This thesis investigates automated Traceability Link Recovery (TLR) for issue--commit linking under realistic repository conditions, with a particular focus on improving precision and reducing the verification burden on developers. The work follows an iterative empirical design in which each experimental step was motivated by limitations observed in the previous one. First, the study revisits the T-BERT baseline by restructuring commit diffs to preserve edit order and change direction, following a future-work direction proposed in prior research. The goal was to test whether a more structured input representation could increase the precision of the model. The results show that these ordered diff variants do not improve performance and slightly underperform the original baseline. Second, after these limitations were attributed in part to the representational and context constraints of the T-BERT architecture, the study evaluates a Small Language Model (SLM) in a zero-shot setting that directly decides whether an issue and a commit are linked. This approach achieves very high recall, but also produces many false positives, resulting in poor precision, especially in larger repositories. Third, Retrieval-Augmented Generation (RAG) is introduced as a repository-grounding mechanism that retrieves semantically related commit fragments as evidence for each issue and injects them into the SLM prompt, while also reducing exhaustive pairwise prompting. Retrieval depth K is treated as a design factor, larger K generally recovers more true links but also increases the number of predicted links, revealing a clear trade-off between recall and verification effort. Overall, the thesis shows that practical traceability link recovery depends not only on model choice, but also on input representation, context limits, calibration, and retrieval design. More reliable automation will require tighter retrieval, stronger filtering, and task-specific adaptation beyond prompt-based inference alone.