Microsoft Claims AI Beats Doctors 83% to 36% on Tough Cases—But There's a Critical Catch in the Comparison

Microsoft has published research in Science claiming its artificial intelligence diagnostic system dramatically outperforms physicians on challenging medical cases—but the comparison may not reflect how healthcare actually works. The company's MAI-DxO system achieved 83% diagnostic accuracy on complex cases compared to just 36% for doctors, a striking gap that Microsoft AI CEO Mustafa Suleyman frames as progress toward "medical superintelligence." The catch? The physicians in the comparison were primary care doctors and general internists working alone, not the specialists or multidisciplinary teams who typically tackle such cases in real-world practice.

How Microsoft's Multi-Agent System Works

The MAI-DxO platform orchestrates multiple AI agents that simulate a panel of doctors collaborating on difficult diagnoses. These agents question patients, order tests, and debate potential diagnoses, mimicking the consultation process that occurs in academic medical centers. Microsoft tested the system on complex case studies from the New England Journal of Medicine, achieving its best results using OpenAI's GPT-5 and GPT-5.5 models.

The underlying AI model's capability proved more important than the orchestration system itself. GPT-5 alone scored 78.9% accuracy on complex cases at approximately $8,400 per case, while the DxO orchestration reduced costs by roughly $3,000 and boosted accuracy by about 4 percentage points. With GPT-5.5, DxO increased accuracy from 72% to nearly 87% at slightly lower cost. However, on emergency department cases, the orchestration provided minimal improvement, with both configurations scoring around 85%.

The Specialist Question and Real-World Testing Ahead

The study's comparison against individual generalist physicians rather than specialist teams represents a significant limitation for assessing clinical utility. Complex cases of the type used in the study would typically be referred to specialists or handled by multidisciplinary teams in actual practice, making the 83% versus 36% comparison potentially misleading about AI's advantage in real healthcare settings.

Microsoft is partnering with Beth Israel Deaconess Medical Center to conduct a real-world clinical evaluation in emergency departments, which should provide more relevant evidence about how AI performs alongside actual clinical workflows. The company is also collaborating with Mayo Clinic on foundation model training, with a release planned for 2027. Dominic King, VP of Health at Microsoft AI, will oversee these clinical validation efforts.

The Race for Medical AI Supremacy

Microsoft's announcement positions the company in direct competition with Google and OpenAI, both of which have published their own claims of doctor-level AI performance. Google previously published its AMIE conversational diagnostic AI study in The Lancet with Beth Israel, while OpenAI has made similar performance claims for its models.

Suleyman predicts healthcare AI models will improve "an order of magnitude" in capability within 24 to 36 months, though Microsoft's own data suggests diminishing returns from orchestration systems as foundation models improve. The computational costs—ranging from $5,400 to $8,400 per complex case—also raise questions about economic viability for widespread deployment, even as the technology shows promise for diagnostic support in academic settings.

The upcoming Beth Israel emergency department study will be critical for determining whether these impressive benchmark results translate into safe, effective clinical decision support in real patient care.

Sources

1. Microsoft's Mustafa Suleyman says its AI outperforms doctors on hard cases as he chases 'medical superintelligence' — TechCrunch