Radiomics performance is inflated by leakage and varies with pipeline choice

Design

Retrospective specification-curve methodological analysis with 456 pipelines per dataset

Population

50 de-identified radiomics classification datasets (median 147 patients) from CT, MRI and PET/CT

Primary outcome

AUC inflation from preprocessing leakage and agreement between near-optimal models

Effect

Pooled AUC inflation 0.09 (95% CI 0.07 to 0.11); near-optimal pipelines disagreed on 20 to 30% of patients

Investigators tested 50 public radiomics datasets covering 13 anatomical sites on CT, MRI and PET/CT, with a median of 147 patients per dataset. They ran 456 analysis pipelines per dataset using repeated cross-validation, and compared leakage-prone preprocessing with a properly nested version.

Oversampling leakage (SMOTE) inflated AUC by a pooled 0.09 (95% CI 0.07 to 0.11), and by more than 0.05 in 64% of datasets. Pipeline choice shifted AUC by 0.30. Pipelines with near-optimal performance disagreed on 20 to 30% of patient classifications and shared fewer than 30% of their top features. Leakage also widened the set of seemingly equally good models fourfold.

The practical message is for readers of radiomics papers: a headline AUC may reflect where preprocessing was done and which pipeline was reported. The authors released an audit tool. These are methodological findings in public datasets, not a verdict on any specific published model.

• Oversampling before cross-validation inflated AUC by a pooled 0.09.
• Near-optimal pipelines disagreed on 20 to 30% of patients.
• Top features overlapped by less than 30% between near-optimal pipelines.
• Datasets were small, median 147 patients.
• Ask whether preprocessing was nested within each validation fold before trusting a radiomics AUC.

Why it matters

It explains why radiomics models that look strong in a paper often fail to carry over to new patients.

Don't overread it

This analysis uses public datasets and shows how methods can inflate results; it does not show that any particular published model is wrong.

The statistics, in plain English

An AUC inflated by 0.09 can turn a model of modest accuracy (for example 0.70) into one that looks good (0.79). Agreement of 70 to 80% between equally good models means about one patient in four or five would be classified differently depending on which pipeline was picked.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.