AI aneurysm detection on CT angiography: more small aneurysms found, at a cost in false alerts

Design

Prospective shadow-mode diagnostic study with adjudication of discordant cases

Population

3,856 consecutive brain CT angiograms at a US health system

Primary outcome

Incremental aneurysm detection and operational metrics

Effect

Sensitivity AI 0.85 (95% CI 0.79 to 0.89) vs radiologist 0.72 (0.65 to 0.78); 55 AI-only true positives, 46 false alerts

In a prospective shadow-mode study at a US health system, a commercial FDA-cleared algorithm read 3,856 consecutive brain CT angiograms while radiologists reported as usual without seeing its output. Disagreements were adjudicated by neuroradiologists.

Agreement was 96.3%. The algorithm was more sensitive than the reporting radiologists (0.85 vs 0.72) with similar specificity (0.99 vs 0.99). It found 55 aneurysms the radiologists missed, against 46 false alerts, while radiologists found 30 the algorithm missed. Most algorithm-only aneurysms were under 3 mm; most radiologist-only finds were 3-5 mm. About 70 scans had to be read for one extra true aneurysm. The yield was best in inpatients (one extra finding per 29 scans) and poorest in outpatients, where false alerts outnumbered extra finds.

The tool and the radiologist miss different aneurysms, which is the argument for combining them rather than replacing one with the other. Many of the extra finds were tiny aneurysms whose clinical significance and management are uncertain, so detection gains need a plan for follow-up, not only a flag.

• Consider AI aneurysm detection as a second reader on CT angiography, especially for inpatient and emergency scans.
• Review each AI flag yourself; about 46 false alerts accompanied 55 extra true finds.
• Look deliberately for small aneurysms under 3 mm, which radiologists missed most often.
• Agree a follow-up pathway for small incidental aneurysms before deploying detection tools.
• Audit local performance by setting; the benefit was smallest in outpatients.

Why it matters

Radiologists and the algorithm missed different aneurysms, which argues for combining them rather than choosing one.

Don't overread it

Only discordant cases were adjudicated, so aneurysms both missed are uncounted, and many extra finds were under 3 mm with uncertain clinical value.

The statistics, in plain English

Sensitivity of 0.85 versus 0.72 means the algorithm detected about 85 of every 100 aneurysms present and radiologists about 72. A gain-to-pain ratio of 1.2 means slightly more true extra finds than false alerts overall. Number needed to examine of 70 means 70 scans run through the tool per extra aneurysm.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.