Impact of Feature Selection Techniques on Classifier Performance : A Multi-Data Experimental Study

Abstract

Feature selection is a prominent challenge in medical machine learning as data pose challenges like imbalance, redundancy, noise, and high dimensionality, making feature selection critical. This thesis systematically compared three feature selection methods ReliefF (filter), LASSO (embedded), and Sequential Feature Selection (wrapper) across six diverse medical datasets using four classifiers Logistic Regression, Random Forest, Support Vector Machine, and K-Nearest Neighbor. Results demonstrated that no single method is universally optimal; instead, its effectiveness depends on dataset characteristics and classifier architecture. Sequential feature selection emerged as the most impactful method overall, particularly, when paired with Random Forest. In three out of six datasets, it achieved best or near-best performance while reducing dimensionality by 40-60%, leading to substantially lowering computational cost and improved accuracy and F1-score. In contrast, LASSO benefits linear classifiers like Logistic Regression and Support Vector Machine as its built-in regularization inherently reduces the influence of irrelevant coefficients. Lastly, ReliefF emphasizes on local interactions and proved effective for datasets with non-linear patterns but provided limited improvement when the original feature space was already relatively small or well-structured. Furthermore, the findings also suggests that feature selection does not consistently lead to performance gains. In certain cases, such as the HCV dataset, using the full feature set produced results comparable to or even better than those from reduced subsets, suggesting that aggressive feature elimination may remove relevant information. In contrast, for datasets with high redundancy, such as ACTG-175, all classifiers showed improvements with Sequential Forward Selection, highlighting that wrapper-based methods can identify compact and highly discriminative feature subsets.

This research offers evidence-based guidance for feature selection methods with multiple classifiers, highlighting that performance is influenced by dataset characteristics, classifier choice, and requirements rather than a single universally optimal method. This work provides a framework for more reliable, efficient, and robust feature selection in machine learning.