The Hidden Dangers Already Lurking in Hospital AI: Dresden Researchers Map the Safety Crisis

Artificial intelligence has quietly entered the clinic through the back door. While healthcare institutions debate formal deployment strategies, large language models are already being used informally by clinicians—a phenomenon researchers call "shadow use." Now, an interdisciplinary team at TUD Dresden University of Technology has published a comprehensive analysis in Nature that maps the full spectrum of risks these systems pose and proposes concrete safety infrastructure to address them.

The timing is critical. According to the research team from the Else Kröner Fresenius Center for Digital Health, only a small proportion of AI systems currently in use have been formally approved as medical devices, creating a gap between rapid clinical adoption and established safety protocols.

Three Categories of Risk Threaten Patient Safety

The Dresden-led study, with first author Dr. Jan Clusmann and senior author Prof. Jakob N. Kather, identifies risks across the entire AI lifecycle. Security vulnerabilities include data poisoning and prompt injections—malicious hidden instructions that could cause an AI system to fail to detect tumors, for example. IT infrastructure weaknesses compound these threats.

Model-inherent safety issues present equally concerning challenges. Large language models are prone to "hallucinations," generating plausible but incorrect medical information that could lead to misdiagnoses. The systems also tend to adapt excessively to user expectations, potentially reinforcing rather than challenging flawed clinical reasoning.

Human-AI interaction risks complete the threat landscape. Clinicians may develop automation bias, over-relying on AI recommendations, or confirmation bias, selectively accepting outputs that match preexisting beliefs. The research team notes that reliability decreases in complex clinical interactions where nuanced judgment is essential.

A Roadmap for Safe Implementation

Rather than simply cataloging dangers, the research team proposes systematic solutions. They recommend that healthcare institutions establish local AI oversight teams to manage deployment and monitor performance. More ambitiously, they call for centralized Security Operations Centers specifically designed for AI systems, providing continuous monitoring and rapid response capabilities.

The study emphasizes the need for updated regulatory frameworks that can keep pace with technology development. Current medical device approval processes were not designed for systems that learn and evolve after deployment, creating regulatory blind spots.

Contributing institutions to the research include University Hospital RWTH Aachen, German Cancer Research Center Heidelberg, National Center for Tumor Diseases Heidelberg, University Hospital Heidelberg, University Medical Center Mainz, Purdue University, University of Pennsylvania, and The Light Collective Eugene.

Bridging the Governance Gap

The Else Kröner Fresenius Center received €40 million in funding from the Else Kröner Fresenius Foundation over ten years, established in September 2019 to advance digital health research.

For healthcare administrators, the message is clear: governance structures must be established now, not after adverse events force reactive measures. For AI developers, the study sets expectations for secure development processes, rigorous data curation, and transparency. For patients, it highlights both the immediate safety concerns and the frameworks needed to ensure human oversight remains central to AI-assisted care.

The gap between AI capability and AI safety infrastructure represents one of healthcare's most pressing challenges. This analysis provides a roadmap institutions can use to address it.

Sources

1. AI in Medicine: Study Weighs Risks, Safe Use Strategies — Mirage News