When AI Gets It Wrong: Why Expert Doctors Catch Errors That Trainees Miss

The promise of artificial intelligence in healthcare rests on a simple assumption: combine algorithmic precision with human judgment, and diagnostic accuracy improves. A new study published in npj Digital Medicine challenges that assumption, revealing that high-performing AI systems may actually undermine the diagnostic skills of trainee clinicians while providing minimal benefit to experienced physicians.

Researchers at Asan Medical Center in Seoul conducted a multi-reader crossover study involving 1,750 interpretation events for neonatal pneumoperitoneum—a time-critical surgical emergency requiring rapid, accurate diagnosis. The findings expose a troubling expertise gap in how clinicians interact with AI recommendations.

The Automation Bias Trap

When trainee clinicians reviewed radiographs aided by high-reliability AI models, they accepted 52.0% of incorrect AI suggestions. This phenomenon, known as automation bias, represents a fundamental challenge to the human-AI partnership model: rather than serving as a safety net, junior clinicians deferred to algorithmic authority even when the AI was demonstrably wrong.

The study's design deliberately tested this vulnerability by exposing clinicians to both high-reliability AI and systematically error-injected models. The contrast in performance was stark. While trainees showed significant susceptibility to automation bias, the three participating neonatologists demonstrated what researchers termed "sentinel behavior"—correctly overriding 91.7% of AI errors (Wilson 95% CI: 83.0–96.0%).

Experts Gain Little, Trainees Learn Less

Perhaps most concerning for healthcare organizations implementing diagnostic AI: high AI reliability offered limited diagnostic gains to expert clinicians. The experts were already performing at such high levels that algorithmic assistance provided minimal incremental value. Meanwhile, the same systems that failed to help experts were actively degrading trainee performance by encouraging over-reliance on automated recommendations.

This creates a paradox for medical education. AI systems designed to support clinical decision-making may inadvertently prevent trainees from developing the adversarial resilience needed to catch algorithmic errors. The study suggests that clinical validity depends on the adversarial resilience of the human-AI team rather than standalone algorithmic performance.

Rethinking AI Deployment and Training

The research has immediate implications for how healthcare organizations validate and deploy diagnostic AI systems. Current regulatory frameworks often require human-in-the-loop validation, but this study suggests that the expertise level of that human matters profoundly. A trainee reviewing AI recommendations may provide less safety oversight than assumed, while an expert may gain little benefit from the system.

In response to their findings, the research team released an open-source educational sandbox designed to train clinicians against automated errors—a recognition that building adversarial resilience requires deliberate practice, not just exposure to high-performing algorithms.

The study, funded by Children's Hospital and Asan Institute for Life Sciences along with Korea Health Technology R&D Project (grant HR20C0026) and the National Research Foundation (RS-2025-00553009), involved five pediatric radiologists and two neonatologists. Their work suggests that the next generation of medical AI must account for expertise-dependent automation bias in both product design and implementation protocols.

Sources

1. Expertise modulates automation bias and sentinel behavior in human–AI collaborative diagnosis of neonatal pneumoperitoneum — Nature Digital Medicine