AI for prostate MRI held up across 12 countries, with site caveats

Design

Retrospective international validation study

Population

1,974 men in external diagnostic and screening cohorts from 12 countries

Primary outcome

Agreement between AI and standard diagnosis of grade group ≥2 cancer

Effect

Agreement 83.0–87.9%; external AUC 0.84–0.89; 28% of sites with AUC below 0.80

SCARLET-1 trained an AI system on 20,471 prostate MRIs from 14 countries and tested it externally on 1,974 men from 20 cities in 12 countries, in one diagnostic and two organised screening cohorts. The target was grade group 2 or higher prostate cancer.

External AUCs were 0.86, 0.89 and 0.84. Agreement with standard-of-care diagnoses was 83.0%, 87.9% and 87.7%, which met prespecified criteria for being interchangeable with an average radiologist. Performance was similar across age, risk, tumour location and ethnicity.

But performance varied considerably by site: 28% of 25 sites had an AUC below 0.80, and image quality mattered (AUC 0.78 to 0.88). The study is retrospective, and agreement with a reference diagnosis is not the same as better patient outcomes.

For departments, this is strong evidence that such tools can generalise, and equally strong evidence that each site must check performance on its own scanners and protocols before relying on one. Image quality remains the radiologist's responsibility.

• Validate any prostate MRI AI tool locally before clinical use
• Audit image quality; AI performance fell with poorer images
• Treat AI output as a second reader, not a replacement, until local performance is known
• Expect variable performance between sites even with the same software

Why it matters

It is among the largest external tests of prostate MRI AI, and it shows site-level variation that summary figures hide.

Don't overread it

Agreement with a reference diagnosis in retrospective data does not show improved outcomes for patients.

The statistics, in plain English

An AUC of 0.86 to 0.89 means the system usually ranked a man with significant cancer above one without. The confidence interval for one screening cohort (0.70 to 0.95) is wide because it had few cancers, so performance there is less certain.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.