AI sepsis prediction: high accuracy on retrospective data, uncertain in new hospitals

Design

Systematic review and meta-analysis of 34 mostly retrospective studies

Population

Hospitalised adults in ML/DL sepsis prediction studies

Primary outcome

Discrimination (AUROC) for predicting sepsis

Effect

Pooled AUROC 0.913 (95% CI 0.887–0.933); prediction interval 0.660–0.983

This systematic review and meta-analysis pooled 34 studies of machine-learning and deep-learning models predicting sepsis in hospitalised adults. Most were retrospective model-development studies, and several reused the same public ICU datasets (MIMIC and PhysioNet), so the 34 reports were not 34 independent populations.

The pooled area under the ROC curve was 0.913 (95% CI 0.887–0.933), but the 95% prediction interval — the range expected in a new setting — ran from 0.660 to 0.983. Models predicting more than four hours ahead performed similarly to those predicting within four hours; the overlap meant no horizon was shown to be better.

The authors are explicit that performance in new clinical populations remains uncertain. Any hospital considering a sepsis alert should ask for local validation and evidence that it changes outcomes, not a published AUROC.

• Ask for local validation data before adopting any AI sepsis alert; a published AUROC does not transfer.
• Check what the model was trained on — many used the same US ICU datasets.
• Measure alert burden and false positives on your own wards during any pilot.
• Do not let an alert replace bedside assessment, lactate and early cultures.

Why it matters

Vendors quote pooled accuracy figures; the prediction interval shows a model could perform only modestly in a new setting.

Don't overread it

A high AUROC does not show that using the model improves patient outcomes — almost none of the included studies tested that.

The statistics, in plain English

The confidence interval (0.887–0.933) is about the average model; the prediction interval (0.660–0.983) is about what you would see in a new hospital. The second, much wider range is the one that matters for anyone buying a model.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.