AI triage tools discriminate well on paper; the evidence for using them is not there yet

Design

Systematic review with meta-analysis and narrative synthesis

Population

27 studies of AI, machine-learning and tele-triage in emergency and acute care

Primary outcome

Discrimination for critical illness and deterioration

Effect

Pooled AUC 0.874 (95% CI 0.845–0.898); I² 94%

This systematic review gathered 27 studies of AI, machine-learning and tele-triage at the first clinical contact in emergency and acute care. Four adult emergency department studies of critical illness or deterioration could be pooled: AI-assisted triage had an area under the curve of 0.874 (95% CI 0.845–0.898), but heterogeneity was extreme (I² 94%). Where studies compared AI with conventional triage directly, AUCs were 0.065 to 0.12 higher with AI.

The review's own warnings are the main point. Most models lacked calibration reporting and external validation, and discrimination does not show better decisions at a real threshold or better patient outcomes. Scenario modelling suggested AI triage might miss 12–38 fewer critical cases per 10,000 visits at the cost of 45–120 extra false escalations. Tele-triage was linked in narrative studies to shorter time to assessment and fewer patients leaving without being seen.

For departments considering these tools, the questions to ask a vendor are how the model performs at your threshold, in your population, and with what false-positive load.

• Ask any AI triage vendor for calibration and external validation in a population like yours.
• Judge a tool at its operating threshold, not by its area under the curve.
• Plan for the extra escalations a more sensitive tool will generate.
• Keep clinician triage as the final decision while tools remain unvalidated.
• Audit missed critical illness locally before and after any change.

Why it matters

AI triage is being sold on discrimination figures that say little about what happens to patients at the front door.

Don't overread it

A pooled AUC from four highly heterogeneous studies is not evidence that AI triage improves outcomes.

The statistics, in plain English

AUC measures how well a model ranks sicker above less sick patients overall; 0.5 is chance, 1.0 is perfect. It does not say how many patients are missed or wrongly escalated at the cut-off actually used. I² of 94% means the studies were measuring quite different things.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.