Healthcare organizations deploying artificial intelligence for clinical decision support may face a critical reliability challenge. A study published in npj Digital Medicine reveals that leading medical large language models exhibit significant reasoning instability, with cancer staging accuracy fluctuating by over 50% based solely on how a question is phrased—even when the underlying clinical information remains identical.

The research, led by Modi, Krull, Gauntner and colleagues, tested three prominent medical LLMs—GPT-5, MedGemma-27B-Text-IT, and OpenBioLLM-Llama3-70B—against 3,035 systematic variations of physician-validated oncology cases, clinical vignettes, and records from the MIMIC database. The findings challenge assumptions that high benchmark scores translate to reliable clinical performance.

Architecture Matters More Than Expected

Using sparse autoencoder analysis trained on 1 billion tokens from 50,000 MIMIC-IV clinical notes, the researchers found that different model architectures encode clinical information in fundamentally different ways. MedGemma exhibits hierarchical encoding, where high-magnitude features capture lexical identity while low-magnitude features encode contextual meaning. OpenBioLLM, by contrast, distributes information uniformly across features.

This architectural divergence has practical implications: interventions or safety guardrails designed for one model architecture may prove ineffective or even counterproductive when applied to another. The study demonstrates that internal encoding structures differentially affect how models retrieve information and perform clinical reasoning tasks.

The models also generated definitive staging recommendations even in scenarios where clinical information was insufficient to support such conclusions—a behavior the researchers note could lead to overconfidence in clinical settings.

Implications for Healthcare Deployment

The findings highlight a gap between laboratory performance and clinical reliability. The 50% accuracy swing from prompt format changes alone represents a consistency challenge that clinicians may not easily detect during routine use.

For health AI companies and hospital systems, the research suggests a need for more rigorous validation protocols. Rather than relying on architecture-agnostic testing, the authors indicate that organizations may need to implement architecture-specific safety validation for each deployed model. What functions as a safeguard for GPT-5 may not translate effectively to MedGemma or OpenBioLLM.

The study, funded by the National Institutes of Health (T32GM139784 and P01CA278732) and the Pelotonia Institute for Immuno-Oncology, arrives as healthcare systems accelerate medical AI adoption. The authors emphasize that mechanistic understanding of how these models reason—not just what they output—may be essential for safe clinical deployment.

Sources

  1. Understanding clinical reasoning variability in medical large language models: a mechanistic interpretability study — npj Digital Medicine