FDA-Cleared AI for Pulmonary Embolism Detection Shows Performance Gap in Multi-Site Real-World Study

When hospitals deploy FDA-cleared artificial intelligence to catch life-threatening blood clots in the lungs, they expect performance matching regulatory approval standards. A new multi-site study reveals a measurable gap between clearance benchmarks and real-world clinical deployment.

Researchers analyzing 67,869 CT scans across 17 healthcare facilities found that two FDA-cleared AI algorithms for pulmonary embolism (PE) detection performed below their regulatory clearance benchmarks in routine clinical use. The findings, published in npj Digital Medicine, document differences between controlled validation studies and real-world deployment that have clinical implications for patient care.

Performance Drops Sharply for Peripheral and Chronic Clots

The Aidoc Medical BriefCase platform—designed to automatically flag pulmonary embolisms for radiologist review—achieved 86.8% sensitivity on dedicated CT pulmonary angiography (CTPA) scans and 73.5% sensitivity for incidental PE detection on routine contrast-enhanced CTs. Both figures fell short of the algorithms' FDA clearance benchmarks, though specificity exceeded cleared levels at 99.1% and 99.8% respectively.

Performance degradation was most pronounced for clinically challenging cases. Sensitivity declined from 99.3% for saddle emboli—large clots blocking the main pulmonary arteries—to 72.9% for subsegmental PE in smaller peripheral vessels. The algorithms also struggled with chronicity: sensitivity dropped from 89.7% for acute clots to 65.3% for non-acute PE, meaning the AI missed more than one-third of chronic emboli.

The retrospective study examined 30,678 CTPA scans and 37,191 routine contrast-enhanced CTs across a 17-facility academic health system. Researchers used a validated large language model pipeline with 97% accuracy to extract reference labels from radiology reports, establishing ground truth for AI performance assessment.

Implications for Post-Market Surveillance

The performance differences documented in this study raise questions about how well controlled validation environments predict clinical deployment performance. While FDA clearance requires demonstration of safety and effectiveness, these real-world results suggest that post-market monitoring may reveal performance patterns not fully captured during pre-market evaluation.

The study received no commercial funding, and while Aidoc Medical provided prediction outputs, the company had no role in analysis or manuscript preparation.

For radiology departments currently relying on AI triage systems for time-sensitive PE diagnosis, these results suggest that continuous monitoring protocols may be warranted—particularly for subsegmental and non-acute emboli where sensitivity dropped below 75%. The diminishing performance for peripheral and non-acute emboli mirrors known human reader limitations, according to the study authors.

The findings underscore the need for standardized post-market surveillance of AI-enabled medical devices. As healthcare systems increasingly integrate commercial AI tools into clinical workflows, transparent real-world performance reporting becomes important for maintaining both clinical trust and patient safety.

The research demonstrates that FDA clearance, while necessary, provides a baseline that may require supplementation with ongoing real-world validation. Healthcare organizations deploying these systems should consider implementing independent validation protocols and performance monitoring to ensure AI tools meet clinical needs across diverse patient populations and imaging conditions.

Sources

1. Multi-site real-world performance of commercial AI for pulmonary and incidental pulmonary embolism detection — Nature