Aneurysm AI on CT angiography: higher sensitivity than radiologists, best value in inpatients

Design

Prospective shadow-mode diagnostic study, radiologists blinded to AI, discordance adjudication

Population

3,856 consecutive brain CT angiograms at a US health system

Primary outcome

Sensitivity, specificity and incremental aneurysm detection

Effect

Sensitivity 0.846 AI vs 0.718 radiologist; 55 AI-only vs 30 radiologist-only true positives

In this prospective shadow-mode study, an FDA-cleared AI algorithm read 3,856 consecutive brain CT angiograms over six weeks while radiologists reported blinded to it. Discordant cases were adjudicated by neuroradiologists.

AI sensitivity was 0.846 (95% CI 0.787-0.894) against 0.718 (0.649-0.779) for radiologists, with similar specificity (0.987 vs 0.985). AI found 55 aneurysms radiologists missed, against about 30 found only by radiologists, and 46 false positives: a gain-to-pain ratio of 1.20 and 70 examinations per extra aneurysm. Yield was best in inpatients (29 examinations per extra find) and unfavourable in outpatients (130 per extra find, more false alarms than gains). Most AI-only aneurysms were under 3 mm.

This goes beyond retrospective accuracy to what a department actually gets and pays in workload. It supports using aneurysm AI as a second reader, particularly for inpatient and emergency work.

The extra finds are mostly tiny aneurysms, many of which will never rupture, so the clinical gain depends on what is done with them.

• Use aneurysm AI as a second reader rather than a triage replacement.
• Consider limiting deployment or alerts in outpatient settings, where false positives outnumbered gains.
• Agree with neurosurgery how incidental aneurysms under 3 mm will be reported and followed.
• Audit your own gain-to-pain ratio after deployment; case mix changes the value.

Why it matters

It gives departments operational numbers, not just accuracy, to decide where AI is worth the extra reads.

Don't overread it

Only discordant cases were adjudicated, so aneurysms missed by both reader and AI are uncounted and sensitivities are overestimated.

The statistics, in plain English

A gain-to-pain ratio of 1.2 means 1.2 true extra aneurysms for each false alarm radiologists had to dismiss. The number needed to examine, 70, is how many scans the AI must read to find one aneurysm a radiologist missed. Because both-missed aneurysms are invisible to this design, the absolute sensitivities are upper estimates, but the comparison between AI and radiologist is fairer.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.