Artificial intelligence tools are being explored for potential clinical applications, but a new comparative study presented at ACR Convergence 2024 in Orlando delivers a sobering reality check: every major AI platform tested made clinically meaningful errors that require physician oversight.

Researchers evaluated five prominent AI platforms—ChatGPT, Gemini, Claude, Perplexity, and OpenEvidence—across 60 rheumatology clinical scenarios, finding that while some tools significantly outperformed others, none proved error-free. The study, led by Dr. Berkha Rani of Mercy Catholic Medical Center in Darby, Pennsylvania, provides comparative benchmark data for AI clinical decision support tools in rheumatology.

Performance Gaps Reveal Stark Differences Between Platforms

The study tested ChatGPT (GPT-5.5), Gemini (Gemini 3), Claude (Opus 4.7), Perplexity, and OpenEvidence against established ACR and EULAR guidelines for six conditions: rheumatoid arthritis, systemic lupus erythematosus, gout, ANCA-associated vasculitis, osteoporosis, and spondyloarthritis.

OpenEvidence emerged as the top performer with 91.2% ± 4.1% guideline concordance and the lowest hallucination rate at just 3.8%. ChatGPT followed closely with 89.4% ± 4.8% concordance and a 6.2% hallucination rate. However, performance degraded sharply from there: Claude achieved 84.7% ± 5.3%, Gemini 78.4% ± 6%, and Perplexity 74.9% ± 6.8% concordance (P < .001).

The hallucination rates—instances where AI confidently generated false or fabricated information—ranged dramatically from OpenEvidence's 3.8% to Gemini's 18.1%. Perplexity registered a 15.6% hallucination rate, while Claude came in at 8.5%.

OpenEvidence scored highest for accuracy at 4.71 on a five-point scale, with ChatGPT close behind at 4.63. ChatGPT distinguished itself by scoring highest for completeness (4.75) and educational value (4.68), while also delivering the fastest response time at 17.5 ± 4.4 seconds compared to the 22-second average across platforms.

Critical Errors Demand Mandatory Human Oversight

The types of errors identified carry serious clinical implications. Researchers documented incorrect medication dosing, outdated therapeutic selections, and inappropriate screening recommendations—clinically meaningful errors that require physician review.

As Dr. Rani emphasized during her presentation, "Fluent is not the same as correct," highlighting a disconnect between AI's conversational confidence and clinical accuracy. Even the best-performing platform made errors in nearly 1 in 10 scenarios, while the worst failed to align with guidelines more than one-quarter of the time.

The researchers noted that all platforms showed "variable but non-negligible limitations." The 13-point spread in hallucination rates between best and worst performers suggests significant variation in how different platforms handle medical evidence.

The Bottom Line for Clinical Practice

The study's central message is clear: no AI platform currently available can be trusted to provide clinical recommendations without physician oversight. Dr. Rani acknowledged that most clinicians are using AI products at home and in the clinic but urged caution: "Performance varies meaningfully by platform. No platform is error free. Clinician oversight is often essential."

Dr. Rani also stressed that the clinical community should take ownership of the development of these platforms to ensure that speed and accuracy benefit clinicians and patients, noting that "evidence-grounded design may matter" and that these models will need to be retested in the future.

As AI continues its exploration in healthcare settings, this research establishes that specialty-specific validation and mandatory human review remain essential safeguards against potentially dangerous AI-generated medical errors.

Sources

  1. 'No platform is error-free': Study of five AI tools shows all require clinician oversight — Healio