A Dataset Generation Method for Bias Evaluation in Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) is a technique generates textual answers on separately retrieved information. While RAG reduces incorrect content in the answers, it has been shown to introduce and amplify biases in model outputs. There is still a lack of dedicated studies and benchmark datasets that systematically investigate how such bias amplification arises and propagates within the RAG pipeline. Drawing on prior work, this paper adopts a preference-based bias measurement framework and introduces a component-aware dataset construction method for datasets used for evaluating bias in real RAG pipelines, and instantiates it for occupation-gender bias using Wikipedia-based knowledge. Our goal is to share the dataset construction methodology, alleviate data scarcity in RAG bias research, and lay a foundation for future studies.