Validation of accurate predictions on small data with a tabular foundation model for clinical decision support

Abstract

Sudden cardiac death represents a leading cause of mortality in chronic heart failure patients. However, the current risk stratification tools demonstrate limited predictive accuracy. Foundation models have gained attention in machine learning through pre-training on large datasets, offering potential advantages for clinical prediction tasks characterised by small sample sizes. TabPFN (Tabular Prior-Data Fitted Networks) extends this paradigm to structured tabular data via in-context learning, potentially enabling improved performance on limited datasets without hyperparameter tuning. Despite promising benchmark results, detailed validation in clinical contexts remains limited, creating a knowledge gap regarding foundation model applicability in medical prediction. To address this gap, this investigation evaluates TabPFN for sudden cardiac death prediction using the MUSIC dataset comprising 992 chronic heart failure patients with 94 sudden cardiac death events. TabPFN performance was compared against three established ensemble methods (Random Forest, XGBoost, CatBoost) through assessment encompassing discrimination, calibration, interpretability, computational efficiency, and robustness evaluation. The study employed appropriate preprocessing pipelines where TabPFN received minimal processing leveraging native categorical handling, while ensemble methods underwent standardisation and class imbalance correction via SMOTE. The results revealed that TabPFN does not achieve expected small-data advantages in this clinical context. Despite statistically significant cross-validation performance relative to base-line methods, TabPFN exhibited notable generalisation challenges comparable to conventional approaches, with all models achieving near-chance independent test performance. Notably, rank reversal occurred between cross-validation and test rankings, revealing that training distribution success does not predict clinical applicability. However, SHAP interpretability analysis demonstrated cross-model consensus in identifying clinically meaningful predictors aligned with established cardiovascular risk factors, providing scientific value despite inadequate discrimination. These findings from the MUSIC dataset contribute to understanding foundation model performance in healthcare contexts. The observed difference between cross-validation results and independent test performance suggests important considerations in model evaluation workflows, highlighting the value of independent test evaluation for clinical machine learning. The study suggests that foundation model advantages observed on benchmark datasets may not transfer to specialised medical predictions task, while the documentation of boundary conditions provides guidance for informed deployment decisions and future architectural development addressing clinical data constraints.