A Comparative Benchmark of Fairness Metrics in Machine Learning
Shahariar, Md Asif (2025)
2025
Master's Programme in Computing Sciences and Electrical Engineering
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2025-12-11
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-2025121011444
https://urn.fi/URN:NBN:fi:tuni-2025121011444
Tiivistelmä
Machine learning is no longer limited to low-risk tasks, but it is also being used in critical decision-making systems. Consequently, machine learning has a direct effect on individuals' lives in domains such as recidivism prediction (COMPAS) and loan approval. Thus, errors or biases generated by such algorithms may have severe consequences in the real world. Therefore, our expectations of ML are not limited to accuracy. In addition to making correct predictions, we expect the models to behave in an unbiased manner toward individuals and groups. Many mathematical definitions of fairness have been proposed by researchers, but there have been few large, systematic comparisons of fairness metrics applied to standard machine learning models. The main task of this thesis is to examine in detail how the balance or trade-off between accuracy and fairness occurs in different models.
The study used two socially sensitive datasets, COMPAS recidivism and Adult Income. These two datasets were used to train and test nine machine learning models. The models used in this study include nine widely used binary classifiers, ranging from simple and interpretable algorithms (e.g., Logistic Regression and Support Vector Machine) to high-performance ensemble methods (e.g., Random Forest, XGBoost, and LightGBM). Each model was carefully evaluated using Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD) which are two important group fairness criteria, and traditional performance metrics including Accuracy and F1-score.
The outcome of the experiment clearly shows a trade-off between accuracy and fairness despite the fact that the nature of this relationship varies greatly among the model architectures. Although ensemble algorithms including XGBoost and LightGBM achieved the highest predictive accuracy, they also had the largest fairness gaps. On the other hand, less complex models like Decision Trees and Multi-layer Perceptrons tended to be more fair and less accurate. One of the key findings in the hyperparameter sensitivity analysis is the fact that fairness is not fixed. It varied with hyperparameters for models like Logistic Regression and SVM, but was less responsive for others. Furthermore, statistical significance testing highlighted that while performance advantages were often robust, differences in fairness rankings were frequently statistically insignificant, cautioning against over-reliance on minor metric variations.
This thesis develops a robust empirical foundation for fairness assessment in machine learning through broad benchmarking with both sensitivity analysis and statistical testing. This study serves as a methodological guide and a source of empirical evidence for practitioners and researchers, providing a framework for evaluating fairness based on evidence-based insights. These results indicate that fairness is not universal or transferable across models or datasets, it must be evaluated contextually, with awareness of model behavior and tuning sensitivity.
The study used two socially sensitive datasets, COMPAS recidivism and Adult Income. These two datasets were used to train and test nine machine learning models. The models used in this study include nine widely used binary classifiers, ranging from simple and interpretable algorithms (e.g., Logistic Regression and Support Vector Machine) to high-performance ensemble methods (e.g., Random Forest, XGBoost, and LightGBM). Each model was carefully evaluated using Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD) which are two important group fairness criteria, and traditional performance metrics including Accuracy and F1-score.
The outcome of the experiment clearly shows a trade-off between accuracy and fairness despite the fact that the nature of this relationship varies greatly among the model architectures. Although ensemble algorithms including XGBoost and LightGBM achieved the highest predictive accuracy, they also had the largest fairness gaps. On the other hand, less complex models like Decision Trees and Multi-layer Perceptrons tended to be more fair and less accurate. One of the key findings in the hyperparameter sensitivity analysis is the fact that fairness is not fixed. It varied with hyperparameters for models like Logistic Regression and SVM, but was less responsive for others. Furthermore, statistical significance testing highlighted that while performance advantages were often robust, differences in fairness rankings were frequently statistically insignificant, cautioning against over-reliance on minor metric variations.
This thesis develops a robust empirical foundation for fairness assessment in machine learning through broad benchmarking with both sensitivity analysis and statistical testing. This study serves as a methodological guide and a source of empirical evidence for practitioners and researchers, providing a framework for evaluating fairness based on evidence-based insights. These results indicate that fairness is not universal or transferable across models or datasets, it must be evaluated contextually, with awareness of model behavior and tuning sensitivity.