The Double-Edged Sword of AI Radiology: Why Catching More Aneurysms Comes at a Cost
A 3,856-patient study reveals that FDA-cleared AI found 39% more brain aneurysms than radiologists but triggered false alarms nearly twice as often, raising critical questions about real-world deployment.
Artificial intelligence is proving it can see what human radiologists miss—but that enhanced vision comes with a significant catch. A large-scale study at Northwell Health has demonstrated that an FDA-cleared AI algorithm detected substantially more brain aneurysms than radiologists, yet simultaneously generated false positives at a rate that could burden clinical workflows and potentially undermine its practical value in certain settings.
The findings, published in the Journal of the American College of Radiology, offer a nuanced portrait of AI's promise and limitations in real-world radiology practice, suggesting that sensitivity gains alone don't tell the complete story of clinical utility.
AI Finds More Aneurysms—But Also More Phantoms
The shadow-mode study analyzed 3,856 CT angiography examinations processed by Aidoc's FDA-cleared algorithm for intracranial aneurysm detection. Operating silently alongside radiologists without influencing clinical decisions, the AI identified 55 confirmed aneurysms that radiologists had missed—a 39% increase in detection. Independent neuroradiologists later adjudicated disagreements to determine ground truth.
However, this enhanced sensitivity came at a price. The AI's positive predictive value reached only 78.2%, compared to 92.7% for radiologists. Of the 101 findings flagged by AI alone, 46 were false positives—meaning clinicians would need to review nearly one incorrect alert for every two genuine aneurysms the system caught. The AI achieved 84.6% sensitivity versus 71.8% for radiologists, while both agreed on more than 96% of examinations overall.
Notably, radiologists also missed cases the AI caught correctly, detecting 30 confirmed aneurysms that the algorithm overlooked, underscoring that neither approach is infallible.
Performance Varies Dramatically by Clinical Context
Perhaps the study's most significant finding is how drastically AI performance shifted across care settings. In inpatient and emergency department environments, the algorithm demonstrated favorable results, identifying 18 additional true aneurysms with only 7 false positives—a ratio that likely justifies the additional review burden in high-acuity settings where missing a diagnosis carries severe consequences.
The outpatient setting told a different story. There, the AI detected just 4 additional true aneurysms while generating more false positives than true positives. This disparity suggests that aneurysm prevalence and patient characteristics significantly influence AI utility, and that blanket deployment across all radiology contexts may not be optimal.
Implications for AI Integration in Clinical Practice
The Northwell Health study illuminates the complex calculus health systems face when implementing AI radiology tools. The technology clearly functions as an effective "second reader" capable of catching smaller or subtler aneurysms that might otherwise go undetected, potentially enabling preventive monitoring before rupture occurs.
Yet the false-positive burden—particularly in lower-prevalence outpatient settings—creates workflow friction that could offset efficiency gains. Radiologists must review each alert, and patients with false-positive findings may face unnecessary anxiety, additional imaging, or clinical follow-up.
For AI developers like Aidoc and competitors in the radiology AI space, the findings underscore that context-specific deployment strategies and potentially adaptive algorithms may be necessary. As health systems move from pilot projects to enterprise-wide implementation, success will likely depend not just on sensitivity metrics but on maintaining high positive predictive value across diverse clinical environments.
The study also raises important questions about incidental findings management and the potential for overtreatment of low-risk aneurysms as detection rates increase—challenges that will require careful clinical protocols as AI radiology tools become more prevalent.
Sources
Companies in this article
Google Search
Follow Health AI Central on Google
Choose Health AI Central as a preferred source, and Google is more likely to show our reporting in Top Stories.
Reporting & accountability
How we maintain accuracy
Health AI Central separates reported facts, analysis, and public-source data. Readers can review our standards, request a correction, or consult the relevant research methodology before relying on a claim.