Healthcare's artificial intelligence revolution faces a sobering reality check this week as two major developments expose critical safety and cost vulnerabilities in clinical AI systems. Doximity's launch of DoseCheck—a verification tool designed to catch AI medication errors—comes alongside a NEJM AI study revealing that large language models systematically recommend care costing more than three times what physicians prescribe, raising urgent questions about AI's readiness for unsupervised clinical decision-making.

The Medication Safety Crisis Hidden in AI Recommendations

Doximity's new DoseCheck tool addresses a troubling discovery: when researchers tested nine widely used AI models with over 8,000 dosing questions, 2.3% of dosing claims were incorrect. More alarming, 43.8% of those detected errors were classified as serious or potentially lethal.

The deterministic, rule-based verification layer works within Doximity Ask to cross-check AI-generated medication dosing against FDA labeling and professional guidelines, increasing dosing accuracy to above 99%. According to Louis-Antoine Mullie, M.D., Doximity's Chief AI Researcher, common error sources include newer drugs, failure to account for patient-specific factors like kidney function, age, and weight, as well as unit or decimal mistakes and drug name confusion.

The findings underscore a fundamental limitation of probabilistic AI models in high-stakes clinical scenarios where precision is non-negotiable. Unlike human physicians who consult reference materials for dosing verification, AI systems may generate plausible-sounding but dangerously incorrect recommendations without built-in safety checks.

The Three-Hundred-Dollar Question: Why AI Overtests and Overrefers

A study published in NEJM AI by researchers from Technion and Stanford University tested 20 general-purpose and four specialized medical AI systems on 200 real patient visit records from the MTSamples database of 4,999 de-identified medical transcription reports. The results revealed a systematic pattern of clinical excess: AI-suggested care averaged $314 per visit compared to $97 for physician-recommended care—a 3.2-fold cost difference.

AI recommendations proved more expensive in 180 of 200 cases (90%). The cost inflation stemmed from two primary sources: diagnostic testing, where AI averaged $176 versus physicians' $71 per visit (a $105 difference), and referrals, with AI recommending 0.87 per visit compared to physicians' 0.15 per visit (creating a $111 cost difference per visit).

Average costs across AI systems ranged from $209 to $599 per visit, and even specialized medical systems showed comparable excess, ranging from 2.7 to 3.5 times physician costs. Researchers attempted cost-aware prompting, which reduced the excess from 3.2 to 2.5 times physician costs. A "parsimonious-but-safe" prompt instructing systems to focus on the three most likely diagnoses and rule out dangerous conditions showed promise but didn't eliminate the cost differential.

Implications for Clinical AI Deployment

The convergence of these findings suggests current AI systems may prioritize comprehensiveness over clinical appropriateness—a tendency that carries both safety and financial consequences. The $217 average cost difference per visit could significantly impact patient out-of-pocket expenses and insurance premiums, while unnecessary diagnostic procedures and specialist referrals expose patients to intervention-related risks without proportional benefit.

For healthcare organizations deploying AI tools, the research indicates that probabilistic models require structured, rule-based safety nets for critical decisions. Doximity's deterministic verification approach offers one model for addressing medication safety, though the broader challenge of clinical judgment—balancing thoroughness against parsimony—may prove more complex to solve through technical guardrails alone.

Meanwhile, Google launched an "AI for Healthcare Professionals Certificate" offering five AMA continuing education credits with completion in under five hours, adopted by institutions including NYU Grossman School of Medicine, University of Alabama at Birmingham, Touro University, and Purdue University School of Nursing. The curriculum is based on AAMC Artificial Intelligence Competencies. Separately, Hims & Hers announced plans to launch Peptides+ specialty following an HHS enforcement discretion policy for six peptides: BPC-157, epitalon, KPV, MOTS-c, semax, and TB-500.

Sources

  1. Comparative Cost of Care Recommended by Large Language Models versus Treating Physicians — NEJM AI
  2. Weekly Rundown: Doximity launches DoseCheck; AI recommends costlier care than doctors, study finds — Fierce Healthcare