Models predicting checkpoint inhibitor pneumonitis show promise but little external validation

Design

Systematic review and meta-analysis of prediction models

Population

12 models in 4,875 lung cancer patients on checkpoint inhibitors (1,057 events)

Primary outcome

Pooled validation C-statistic for pneumonitis occurrence

Effect

C-statistic 0.82 (95% CI 0.75 to 0.88), I-squared 90.7%, prediction interval 0.51 to 0.95

Checkpoint inhibitor pneumonitis is a potentially fatal immune-related adverse event in lung cancer, and many machine learning, CT radiomics and clinical models claim to predict it. This systematic review pooled 12 peer-reviewed models covering 4,875 patients and 1,057 events.

The pooled validation C-statistic was 0.82 (95% CI 0.75 to 0.88), with high heterogeneity (I-squared 90.7%). The 95% prediction interval, which the authors treat as the decision-relevant figure, was wide at 0.51 to 0.95. Internal cross-validation gave 0.83 and internal split-sample 0.76; only two studies had external validation (0.84 and 0.92, described narratively). Imaging models (8) had numerically higher discrimination than clinical models (4), 0.85 against 0.78, without a significant difference (p = 0.46). Risk of bias was high in two studies, unclear in nine and low in one, and certainty of evidence was low.

The authors conclude that the models separate risk groups on average but are unready for clinical use. The prediction interval reaching 0.51 means a new model in a new centre could perform no better than chance.

• Pooled validation C-statistic was 0.82, but the 95% prediction interval ran from 0.51 to 0.95.
• Only two studies had external validation.
• Imaging models were not significantly better than clinical ones (0.85 vs 0.78, p = 0.46).
• Do not adopt a published radiomics model for pneumonitis risk without local validation.
• Keep baseline and follow-up chest CT review focused on early signs of pneumonitis.

Why it matters

Radiology departments are being offered such tools, and the evidence does not yet support relying on them.

Don't overread it

Low-certainty evidence and mostly internal validation mean the pooled figure probably overstates real-world performance.

The statistics, in plain English

A C-statistic of 0.82 means the model ranked a patient who developed pneumonitis above one who did not about 82% of the time on average. The prediction interval of 0.51 to 0.95 shows how much a single new application could vary, from useless to excellent.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.