Machine learning models for diabetic kidney disease look accurate but are rarely validated

Design

Systematic review and bivariate meta-analysis of prediction models

Population

66 machine learning models for diabetic kidney disease

Primary outcome

Discrimination (AUC), sensitivity and specificity

Effect

AUC 0.93 (0.90–0.95); I² >85%

A systematic review pooled 66 machine learning models built to predict or diagnose diabetic kidney disease. The pooled AUC was 0.93 (95% CI 0.90–0.95), with sensitivity 0.83 and specificity 0.90.

Heterogeneity was very high (I² above 85%), external validation was rare, calibration was seldom reported and methodological transparency was poor. Model type and study design both modified the apparent performance.

High AUCs from internally tested models usually shrink in new populations. Albuminuria and eGFR remain the tests to use, and none of these models is ready to replace them.

• Continue to screen people with diabetes for kidney disease with urine albumin-creatinine ratio and eGFR.
• Ask for external validation and calibration data before adopting any prediction model.
• Be sceptical of very high AUCs from models tested only in the data that built them.
• Check whether a model was developed in a population similar to your own.

Why it matters

A model that has only been tested on its own development data tells you little about your patients.

Don't overread it

Pooled accuracy here reflects largely internal validation and is likely optimistic.

The statistics, in plain English

An AUC of 0.93 looks excellent, but I² above 85% means the studies disagree widely, and without external validation the true performance in new patients is unknown.

Read the rest in the app

You have read your two free briefings this month. The app carries all 27 specialties, every morning, free — and this finding is waiting in it.

Get Daily Doctor — free

Scan to keep reading on your phone. No account needed to start.