Design: Systematic review and meta-research analysis
Population: 461 records: 68 trial reports, 15 protocols and 378 registrations of LLM interventions
Primary outcome: Completeness of reporting against matched guidelines
Effect: Mean completeness 40–59% for reports, 51–57% for protocols, 61% for registrations

A BMJ systematic review examined 461 records of clinical trials testing large language model interventions — 68 trial reports, 15 protocols and 378 registrations up to December 2025. Most trials were small, single-site and published in 2024 or 2025, and most tested patient education or management advice.

Reports met between 40% and 59% of the items in established reporting standards, depending on the checklist. The details most often missing were the ones specific to these tools: which model version was used, how it was accessed and set up, how prompts and outputs were processed, and how harms were planned for and monitored.

For clinicians being offered chatbot-based tools, this matters directly. A positive trial that does not say which model it tested cannot tell you whether the product in front of you is the same thing.

Ask which model version and settings a vendor's evidence actually tested
Check whether a trial of an AI tool planned and reported harms, not just benefits
Treat a small single-site positive trial of a chatbot as early evidence
Remember that models change after a trial ends, so results may not carry forward

Why it matters

Hospitals are buying these tools now, on evidence that often cannot be reproduced.

The statistics, in plain English

Reporting completeness is the share of checklist items a paper reported. A score of 44% means more than half of the information a reader needs to judge bias or repeat the trial was missing.