When a new tool reports very high accuracy, check what was compared. Ask whether the cases were consecutive patients from the real referral pathway or a selected set, and what the reference standard was.
A tool tested on clear-cut cases and healthy controls will look better than it does in the population where doubt actually arises.
- Check whether the study cohort resembles your own referral population.
- Look at how the reference standard was obtained, and in whom.
- Ask whether comparison readers were given the same information as the tool.
- Prefer results from external, prospective validation.
Why it matters
Selection and reference standard often explain more of a result than the method itself.
Read the rest in the app
You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.
Scan to keep reading on your phone. No account needed to start.
Tomorrow morning, before your first patient
One edition a day for radiology, written by the desk, every claim tied to its paper. Six minutes.
Get the app — free