A suicide prediction model built on 1.9 million assessments still misses most cases

Design

Prognostic machine-learning study with a held-out test sample

Population

668,684 regular US Army soldiers; 1,932,243 health assessments, 2015 to 2019

Primary outcome

Suicide and nonfatal suicide attempt within 24 months

Effect

AUROC 0.72 (suicide) and 0.81 (attempt); top 5% risk band captured 18.6% of suicides

US researchers used administrative data from about 1.9 million periodic health assessments of regular Army soldiers (2015 to 2019) to build machine-learning models for suicide and for nonfatal attempts over 24 months. Suicide prevalence was 66.7 per 100,000; attempt prevalence was 704.1 per 100,000.

In the held-out test sample the suicide model had an AUROC of 0.72 and the attempt model 0.81. Suicide risk was meaningfully raised, at least twice chance, only in the top 5% of predicted risk, and that band captured 18.6% of suicides. For attempts, risk was raised in the top 10%, capturing 46.5%.

Predicted risk of suicide and of attempt correlated only modestly (r = 0.29), so the two have distinct predictors. For clinicians, the lesson is that even a very large model leaves most suicides outside the high-risk group. Population-level targeting may help allocate a skills-training programme, but it cannot clear an individual patient.

• Do not treat a low calculated risk as reassurance for an individual patient.
• Use clinical assessment of intent, plan and access to means at every contact.
• Remember that risk factors for attempts differ from those for death by suicide.
• Offer brief structured interventions, such as safety planning, broadly rather than only to flagged patients.

Why it matters

It challenges the assumption that more data and machine learning will let clinicians reliably single out who will die by suicide.

Don't overread it

This was a prognostic study in a military population and does not test whether using the model saves lives.

The statistics, in plain English

Sensitivity of 18.6% means that four in five suicides occurred among people the model did not flag as high risk. An AUROC of 0.72 is only modest discrimination, and the model was built in a US military population, so it may not transfer to other settings.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.