Why Top Medical AI Models Fail Half of Real Clinical Tasks Despite Acing Licensing Exams

Medical AI systems can pass licensing exams with flying colors, but when faced with the messy reality of clinical workflows—navigating electronic health records, coordinating multi-step tasks, and executing administrative processes—they fail more than half the time. That's the sobering conclusion from Stanford HAI researchers who spent three years developing progressively realistic benchmarks to measure what medical AI can actually accomplish in healthcare settings.

The findings reveal a "generative AI chasm" where models excel at generating medically accurate text but stumble when required to execute complete workflows across real healthcare systems. The best-performing model, GPT-5.5, completed only 46% of clinical tasks on a single attempt, with consistency dropping to just 28% across three runs. On administrative tasks—the workflows consuming over $1 trillion annually in U.S. healthcare—performance fell to 36.3% completion.

From Text Generation to Real-World Execution

Stanford's research team developed three benchmark systems that progressively mirror actual healthcare work. MedHELM, created in 2024 with input from 29 clinicians, established a taxonomy spanning 5 categories, 22 subcategories, and 121 tasks across 37 datasets. The evaluation of 9 frontier models revealed significant performance variation: while clinical note generation achieved win rates of 0.74-0.85 and patient communication scored 0.76-0.89, administration and workflow tasks languished at 0.53-0.63.

PhysicianBench took realism further with 100 long-horizon tasks derived from actual patient consultations across 21 specialties. These tasks averaged 27 tool calls each and included 670 evaluation checkpoints. The results exposed a critical failure pattern: 62% of GPT-5.5's action-execution errors involved writing clinically correct plans but never actually placing orders in the EHR system—a coordination failure that would render the AI ineffective in practice.

The Administrative Bottleneck

HealthAdminBench targeted administrative workflows where staff may outnumber clinicians 10:1. The benchmark included 135 tasks across prior authorizations, appeals and denials, and durable medical equipment orders, evaluated through 1,698 verifiers combining deterministic checks and LLM-judge rubrics.

Claude Opus 4.6, the top performer on administrative tasks, achieved an 82.8% subtask pass rate but only 26.7% whole-task completion—demonstrating that AI can handle individual steps but fails to orchestrate complete workflows. Open-source models lagged further behind, with DeepSeek V4-Pro managing just 19% clinical task completion.

The research team's MedAgentBench, featuring 300 patient-specific tasks across 100 synthetic profiles, showed rapid improvement from under 70% completion at its 2025 release to over 90% by 2026, suggesting the field is advancing but still faces fundamental execution challenges.

Implications for Healthcare Organizations

These findings carry implications for health systems evaluating AI investments. The highest-value use cases—prior authorizations, specialist consultations, and administrative workflows—are precisely where current AI performs worst, according to the Stanford research. Healthcare organizations may need to shift procurement criteria from text-generation demonstrations to execution-grounded evaluations that measure end-to-end task completion in realistic workflow scenarios.

For AI developers, the benchmarks provide a potential improvement roadmap: solving coordination failures between planning and execution, maintaining persistent memory across workflow steps, and achieving reliable action execution within EHR environments. The gap between vendor promises and operational reality remains wide, but these benchmarks provide measurement tools to assess performance.

Sources

1. Measuring What Medical AI Can Actually Do — Stanford HAI