A groundbreaking clinical trial has delivered a sobering reality check for artificial intelligence in medical diagnosis: while AI can dramatically accelerate clinical decision-making, it may not actually help doctors get the right answer—and might even make them overconfident in wrong ones.

The ALLIANCE trial, presented at ACR Convergence in Orlando, represents the first multicenter randomized controlled study of a medically certified large language model used as a diagnostic support tool in rheumatology. The results challenge the assumption that faster AI assistance automatically translates to better patient care.

Speed Without Precision: The Core Findings

Eighty-two physicians from seven hospitals—including 21 rheumatologists and 61 non-rheumatologists—evaluated case vignettes of three challenging rheumatologic conditions: Cogan syndrome, dermatomyositis, and familial Mediterranean fever. Half received LLM support, while the control group used conventional resources.

The speed advantage was dramatic: physicians using the LLM completed their diagnostic assessments in just 94 seconds on average, compared to 206 seconds for those using traditional resources—a 54% reduction in processing time.

However, diagnostic accuracy told a different story. Post-intervention top-one diagnostic accuracy reached 33.3% in the LLM group versus 35% in controls—a difference that was not statistically significant (adjusted OR = 0.99; 95% CI, 0.45 to 2.19). Both groups started from similarly low baselines of 22.2% and 23.3% accuracy, respectively.

Even when researchers examined top-three diagnostic accuracy—whether the correct diagnosis appeared among a physician's three most likely candidates—the LLM group showed only a numerically higher rate of 49.2% versus 39.2% for controls, which failed to reach statistical significance (P = .291).

The Overconfidence Problem

Perhaps most concerning was a pattern of overconfidence observed across all settings that actually increased following the intervention in both groups. This suggests that access to rapid information—whether from AI or conventional sources—may inflate clinicians' certainty in their diagnoses without improving their actual accuracy.

"This overconfidence is particularly troubling," noted Johannes Knitza, MD, PhD, senior consultant rheumatologist and deputy head of the Institute for Digital Medicine at Marburg University Hospital in Germany, who led the research. The findings underscore the need for physicians to maintain critical evaluation of AI suggestions rather than developing over-reliance on these tools.

Despite the accuracy limitations, physicians in the LLM group reported positive experiences with the system, noting more timely information delivery and higher perceived quality. Participants indicated they liked the system and would use it again.

Implications for Clinical AI Development

The ALLIANCE trial provides crucial evidence for healthcare stakeholders navigating the AI revolution. For developers, it demonstrates that regulatory certification and workflow efficiency alone don't guarantee clinical value—accuracy enhancement and proper calibration of clinician confidence remain essential targets.

The researchers acknowledged that testing only three cases represents a limitation and have planned follow-up studies to expand the evidence base. Nevertheless, this first randomized controlled trial of a CE-certified LLM-based clinical decision support system in rheumatology establishes an important benchmark for evaluating future AI diagnostic tools.

Sources

  1. Large language model improves speed but not accuracy of rheumatology diagnoses — Healio