- Design
- Prospective intra-individual reader study with eye tracking
- Population
- 10 radiology residents, 33-film chest radiograph sets
- Primary outcome
- Diagnostic accuracy vs expert consensus (kappa) and reporting time
- Effect
- Fatigued readers kappa 0.71 to 0.61 (P below 0.001); AI change 0.03 to 0.04, not significant
A prospective reader study asked ten radiology residents to interpret sets of 33 chest radiographs before and after a first-call on-site night shift, with and without AI assistance. Eye tracking measured oculomotor efficiency, and fatigue was defined as a drop of at least 5% in a standard saccade measure.
Six of the ten met the fatigue definition. Their agreement with expert consensus fell from kappa 0.71 to 0.61 after the shift; non-fatigued readers dropped less, from 0.72 to 0.65. Reporting time per film did not change. The AI tool did not significantly improve accuracy (kappa change 0.03 to 0.04), though it shortened reporting time in non-fatigued readers.
The worrying finding is the combination: accuracy fell while speed held, so neither the reader nor a supervisor would notice from throughput alone. The AI result applies to this one model and should not be generalised.
The study is small and used a curated film set, so the size of the effect in live practice is uncertain. It still supports protecting post-night-shift reporting and building in second reads for high-stakes films.
- Avoid scheduling high-stakes reporting immediately after a night shift where rotas allow.
- Do not assume steady reporting speed means steady accuracy.
- Consider second reads of overnight chest films flagged as significant.
- Do not rely on this AI tool to compensate for fatigue.
Why it matters
A tired reader may not feel slower, and their output will not look slower either.
Don't overread it
Ten residents reading a curated set; this does not measure error rates in live reporting.
The statistics, in plain English
Kappa measures agreement beyond chance; a fall from 0.71 to 0.61 moves from substantial to moderate agreement. With only ten readers, the estimates are imprecise.
Read the rest in the app
You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.
Scan to keep reading on your phone. No account needed to start.
Tomorrow morning, before your first patient
One edition a day for radiology, written by the desk, every claim tied to its paper. Six minutes.
Get the app — free