Design: Benchmark evaluation of six commercial large language models
Population: 225 clinical scenarios from 15 Choosing Wisely recommendations; 4050 responses
Primary outcome: Overuse rate and false-restraint rate
Effect: Overuse 5.4% (0.6–12.2%); false restraint 0.6% (0.1–1.2%); thyroid ultrasound overuse 42.8%

A benchmark study in Annals of Internal Medicine, published on 20 October, built 225 scenarios from 15 Choosing Wisely recommendations, some where restraint was right and some where action was. Six commercial large language models answered each three times, giving 4050 responses.

Overall, the models recommended low-value care 5.4% of the time and wrongly held back appropriate care 0.6% of the time. Overuse clustered in two areas: thyroid ultrasound for a small incidental nodule (43% of responses) and routine ECGs in low-risk adults (28%).

Clinicians increasingly ask these tools for advice. The study suggests they are generally restrained but share the same habits that drive overuse in practice. It was a single-prompt benchmark, not real use, and a model judged the answers with physician checking.

Check AI-generated suggestions for imaging of small incidental thyroid nodules; they often overcall it.
Do not accept an AI suggestion of a routine ECG in a low-risk, asymptomatic adult without a reason.
AI tools rarely withheld appropriate treatment in this benchmark.
Results came from structured scenarios and may differ in real conversations.

Why it matters

As clinicians consult language models, the models' bias towards testing becomes the clinician's.

The statistics, in plain English

The overall overuse rate of 5.4% had a wide confidence interval (0.6% to 12.2%) because it varied so much between recommendations. The average hides that two scenarios accounted for most of it.