Models predicting immunotherapy pneumonitis look good on average but are not ready for the clinic

Design

Systematic review and meta-analysis of prediction model validation studies

Population

12 models, 4875 lung cancer patients on immune checkpoint inhibitors, 1057 pneumonitis events

Primary outcome

Validation C-statistic for predicting checkpoint inhibitor pneumonitis

Effect

Pooled 0.82 (95% CI 0.75 to 0.88); prediction interval 0.51 to 0.95; I² = 90.7%

Checkpoint inhibitor pneumonitis can be fatal, and many CT radiomics, deep-learning and clinical models claim to predict it. A systematic review pooled 12 models for lung cancer patients, covering 4875 patients and 1057 pneumonitis events.

The pooled validation C-statistic was 0.82 (95% CI 0.75 to 0.88), but heterogeneity was very high (I² = 90.7%) and the 95% prediction interval ran from 0.51 to 0.95 — meaning a model applied in a new hospital could perform anywhere from no better than chance to excellent. Only two models had external validation. Imaging models scored numerically higher than clinical ones (0.85 vs 0.78), a difference that was not significant. Most studies had unclear risk of bias and certainty was low.

For radiologists, the value remains in recognising pneumonitis patterns on CT in patients on immunotherapy, not in a predictive score.

• Do not use published radiomics or AI models to predict checkpoint inhibitor pneumonitis in routine reporting.
• Note whether a patient is on immune checkpoint inhibitors when reporting new ground-glass or organising pneumonia patterns.
• Compare with the pre-treatment baseline CT when new parenchymal change appears during immunotherapy.
• Ask for treatment history on the request if it is missing; it changes the differential.

Why it matters

A good average C-statistic can hide a model that fails in a new hospital, as the wide prediction interval shows.

Don't overread it

The pooled 0.82 does not mean any one model will perform that well locally; only two were externally validated.

The statistics, in plain English

A C-statistic of 0.5 is chance and 1.0 is perfect. The confidence interval describes uncertainty about the average model; the much wider prediction interval (0.51 to 0.95) describes how a model might perform in a new setting, which is what matters when deciding whether to use it.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.