What the AI is actually marking when it flags a normal screening mammogram

Design

retrospective informed radiological review of selected cases from a national screening programme

Population

130,031 screening mammograms from 42,371 women, BreastScreen Norway 2008-2018; 240 reviewed in two samples of 120

Primary outcome

mammographic features and density associated with the highest 5% of AI risk scores

Effect

in high-score cancer-free cases, calcifications alone in 72% and 68% for the two models, 76% read as benign; in high-score screen-detected cancers, spiculated mass most frequent at 29%

Two commercial AI models were applied to 130,031 screening mammograms from 42,371 women in BreastScreen Norway. Radiologists then reviewed, with knowledge of the AI output, two samples drawn from the highest 5% of risk scores by both models: 120 mammograms with a high score and no breast cancer over the following six years, and 120 with a high score and a screen-detected cancer.

The two groups looked different. In the high-score cancer-free group, calcifications alone were the dominant marked feature - 72% for one model, 68% for the other - mostly amorphous in morphology and clustered in distribution, and 76% of these had been given the lowest radiological interpretation score, meaning the reading radiologist had considered them benign. Dense breasts were over-represented in this group as well (BI-RADS d in 11% against 3%). Among the screen-detected cancers, the commonest feature was a spiculated mass, at 29%.

The practical value is in knowing where an AI flag is least informative. A high score driven by amorphous clustered calcifications that a radiologist has already judged benign is the pattern most likely to be a false positive; a high score over a spiculated mass is the pattern that most often is not. That is a reading behaviour rather than a rule, and it is worth having before an AI tool arrives in a department, because the first weeks of use are when a recall rate can be pushed up without anyone intending it.

• When an AI flags calcifications alone that you have already assessed as benign, weight your own assessment - that is where false positives concentrate
• Give an AI flag over a spiculated mass more weight than one over calcifications or density
• Expect more high scores in dense breasts and do not let that drive recall on its own
• Audit recall rate for the first months after introducing an AI tool, against the period before
• Agree in advance what a radiologist does when they disagree with the AI, and record the disagreement

Why it matters

It tells a reader which AI flags to argue with, which is the part of AI-assisted screening that determines the recall rate.

Don't overread it

A descriptive review of selected cases with informed reviewers - it does not measure how accurate either model is.

The statistics, in plain English

This is a descriptive review of 240 selected mammograms, not a test of accuracy - there is no sensitivity, specificity or predictive value here, and none can be read off it. What it gives is the composition of the false positive group: knowing that roughly 70% of the non-cancer high scores were calcifications tells you what to be sceptical about, but not how often an AI flag on calcifications is wrong overall. The six-year cancer-free window is a genuine strength; a shorter follow-up would have misclassified slow-growing cancers as false positives.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.