When Doctors Override AI: The Equity Gap Healthcare Algorithms Can't Fix Alone

The healthcare industry has invested heavily in creating mathematically fair AI algorithms, but a provocative new perspective suggests the real source of bias may lie not in the code, but in the exam room. Researchers from NYU Grossman School of Medicine argue that even perfectly calibrated algorithms can produce inequitable outcomes when clinicians apply AI recommendations differently based on patient demographics and social status.

Writing in the Journal of General Internal Medicine, Kellie Owens, Maggie Ramatowski, and Saul B. Blecker contend that the field's focus on algorithmic fairness metrics has overlooked a critical vulnerability: how physicians actually use AI outputs in clinical practice. Their analysis suggests that marginalized patients may receive less individualized clinical judgment when doctors accept AI recommendations at face value, while socially advantaged patients benefit from greater scrutiny and personalized interpretation of the same algorithmic outputs.

The Implementation Blind Spot

Current approaches to healthcare AI fairness emphasize distributive justice metrics—ensuring algorithms deliver consistent sensitivity and positive predictive value across demographic groups. But the NYU team argues this framework ignores how clinicians interpret and act on AI recommendations in real-world settings.

The authors report being unaware of studies directly examining bias in AI implementation within healthcare, though related clinical decision support literature reveals effectiveness differences across gender and age groups. This evidence gap is particularly concerning given the rapid deployment of AI tools across health systems.

The researchers identify several mechanisms through which implementation bias emerges. Automation bias—the tendency to defer to outputs perceived as objective—may lead clinicians to accept AI recommendations without adequate scrutiny. Limited AI literacy compounds the problem, producing both overreliance and unwarranted skepticism depending on clinician training and experience. Decision fatigue and alert burden in high-pressure clinical environments may further reduce the individualized attention given to AI-generated recommendations.

From Human-in-the-Loop to System-in-the-Loop

The traditional "human-in-the-loop" model positions clinicians as a safety mechanism to catch algorithmic errors. But Owens and colleagues challenge this assumption, suggesting clinicians may themselves be "the very conduit through which inequities enter the system" rather than a fail-safe.

Their proposed solution shifts from individual oversight to organizational accountability through "system-in-the-loop" governance. This framework requires healthcare organizations to monitor clinician behavior patterns across patient populations, tracking override rates, documentation practices, and downstream outcomes by demographic subgroups.

For health systems, this means implementing new infrastructure including audit dashboards, standardized documentation requirements when overriding AI recommendations, and workflow designs that account for cognitive load. For AI vendors, responsibilities extend beyond model deployment to providing tools for continuous subgroup performance monitoring and tracking clinician interaction patterns.

Implications for Healthcare Equity

The analysis suggests marginalized populations face a "double jeopardy" where both algorithmic limitations and differential clinician interpretation compound existing healthcare inequities. While advantaged patients may benefit from physicians who critically evaluate and personalize AI recommendations, disadvantaged patients may receive more algorithmic and less individualized care.

This framework has immediate implications for the hundreds of AI clinical decision support tools currently deployed across health systems. Organizations must move beyond validating algorithms at deployment to continuously monitoring how clinicians use these tools in practice—a shift that requires substantial investment in governance infrastructure and cultural change around AI accountability.

The research, published with DOI 10.1007/s11606-026-10767-y, calls for a fundamental reframing of healthcare AI equity efforts, recognizing that fair algorithms are necessary but insufficient for equitable care delivery.

Sources

1. The Hidden Bias in AI Healthcare May Come From Clinicians, Not Algorithms — bioengineer.org