An AI prostate MRI reader held up across 12 countries, but not at every site

Design

Retrospective multinational validation study

Population

1,974 external patients from 20 cities in 12 countries; 20,471 MRIs for training and internal testing

Primary outcome

Agreement with standard-of-care diagnosis of grade group ≥2 cancer

Effect

External AUC 0.84–0.89; agreement 83.0–87.9%, meeting interchangeability criteria

SCARLET-1 trained an AI system on over 20,000 prostate MRIs from 14 countries and tested it on 1,974 patients from 20 external cities in 12 countries, in one diagnostic and two organised screening datasets. The target was Gleason grade group 2 or higher cancer.

External AUCs were 0.86, 0.89 and 0.84. Agreement with standard-of-care diagnosis was 83–88%, meeting the prespecified test of interchangeability with an average radiologist in all three. Performance was stable across age, risk, tumour location and ethnicity.

It was not stable across sites: 28% of 25 sites had an AUC under 0.80, and image quality moved AUC between 0.78 and 0.88. Any department adopting such a tool needs to validate it on its own scanners and protocols, and image quality matters as much as the algorithm.

• Validate any prostate MRI AI tool on local scanners before relying on it.
• Audit image quality, since AI performance fell with poorer images.
• Keep a radiologist read; this was a retrospective validation, not a prospective trial.
• Track site-level performance rather than relying on a vendor's pooled figure.

Why it matters

A good average can hide a poor result at your own site.

Don't overread it

Retrospective validation does not show how the tool performs inside a live reporting workflow.

The statistics, in plain English

An AUC of 0.86 means the system ranks a random cancer above a random non-cancer 86% of the time. The wide interval in the smallest screening set (0.70–0.95) reflects few cancers.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.