Comparative Analysis of AI-Augmented vs. Rule-Based SAST in Identifying Vulnerabilities
Alhourani, Ridvan (2026)
2026
Tietotekniikan DI-ohjelma - Master's Programme in Information Technology
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2026-06-16
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202606117287
https://urn.fi/URN:NBN:fi:tuni-202606117287
Tiivistelmä
The primary problem this study addresses is the need for a Static Application Security Testing (SAST) tool that is easy to integrate into a DevSecOps pipeline while maintaining a low rate of false positives. Although various solutions for high false-positive rates have been attempted over the years, the field continues to evolve. Recent studies have utilized Large Language Models (LLMs) and Artificial Intelligence (AI) to address this issue with promising results; however, in this study and within the time of its processing; these solutions proved to be complicated and costly for individuals and small-sized teams to implement.
This gap led to the central research question of whether an accessible, AI-augmented SAST tool exists in the market for such users. To investigate this scientifically, this study compares a traditional rule-based tool, SonarQube, with an AI-augmented SAST tool, Snyk against the vulnerability information described in the vulnerable OWASP Juice Shop app’s documentation.
The investigation focused on verifying the tools' results in terms of code coverage, accuracy of findings, and vulnerability categorization accuracy. The author manually examined 558 findings to assess whether the tools demonstrated contextual awareness and to evaluate the quality of the remediation information provided. An important contribution of this research is the creation of a manually verified dataset of these findings, categorized by validity and utility, which serves as a point of reference for future security tool evaluations.
The investigation concluded that the AI-augmented tool, Snyk, excelled in accuracy, contextual awareness, and the quality of its remediation guides. On the contrary, while SonarQube's coverage exceeded Snyk's by a small margin, it lagged in terms of accuracy and the utility of the remediation steps provided. Specifically, the manual verification revealed that 73.4% of SonarQube’s findings were false positives, compared to a 77.9% false-positive rate for Snyk. However, Snyk demonstrated superior contextual relevance in its valid findings. Finally, this study found that SAST tools alone are insufficient due to the nature of certain flaws that require alternative analysis methods, such as Dynamic Application Security Testing (DAST) or manual penetration testing.
This gap led to the central research question of whether an accessible, AI-augmented SAST tool exists in the market for such users. To investigate this scientifically, this study compares a traditional rule-based tool, SonarQube, with an AI-augmented SAST tool, Snyk against the vulnerability information described in the vulnerable OWASP Juice Shop app’s documentation.
The investigation focused on verifying the tools' results in terms of code coverage, accuracy of findings, and vulnerability categorization accuracy. The author manually examined 558 findings to assess whether the tools demonstrated contextual awareness and to evaluate the quality of the remediation information provided. An important contribution of this research is the creation of a manually verified dataset of these findings, categorized by validity and utility, which serves as a point of reference for future security tool evaluations.
The investigation concluded that the AI-augmented tool, Snyk, excelled in accuracy, contextual awareness, and the quality of its remediation guides. On the contrary, while SonarQube's coverage exceeded Snyk's by a small margin, it lagged in terms of accuracy and the utility of the remediation steps provided. Specifically, the manual verification revealed that 73.4% of SonarQube’s findings were false positives, compared to a 77.9% false-positive rate for Snyk. However, Snyk demonstrated superior contextual relevance in its valid findings. Finally, this study found that SAST tools alone are insufficient due to the nature of certain flaws that require alternative analysis methods, such as Dynamic Application Security Testing (DAST) or manual penetration testing.