When Six AI Systems All Pick the Same \"Random\" Number: What Clinical Trials Need to Know About Machine Confidence

When Mohanish Anand, a Global Trial Management lead, asked six major AI models to randomly select a number between 1 and 30, every single system chose 17. More troubling than the contradictory explanations each provided for why 17 was "random," this deceptively simple experiment has become a rallying point for clinical trial experts warning that AI's fundamental problem isn't what it can't do—it's that these systems produce confident, plausible outputs based on historical patterns while lacking genuine reasoning about current contexts.

The implications extend far beyond parlor tricks. As AI tools proliferate across clinical trial operations—from protocol deviation flagging to site risk scoring—industry leaders are sounding alarms about a systemic honesty problem that threatens patient safety, data integrity, and healthcare equity.

The Automation Bias Trap in Trial Oversight

Doug Bain, a consulting partner in trial technology, crystallizes the core risk: "If your AI makes 10,000 decisions and a human clicks Approve 10,000 times, you haven't necessarily created a control. You may just have created a very expensive button." This phenomenon, known as automation bias, describes a documented cognitive tendency where humans accept machine output without independent verification—particularly when machines rarely appear wrong.

Rudy Malle, a clinical research professional trainer, identified critical capabilities AI cannot independently own: patient safety monitoring, data integrity verification, risk interpretation, site coaching, exception management, and regulatory accountability. Yet high-volume AI-assisted approval workflows risk transforming substantive human oversight into rubber-stamping exercises, creating the illusion of control while systematically undermining it.

The challenge isn't technological sophistication—it's institutional discipline. Subhajit Sengupta, Associate Director of Data Science at Cytel, argues AI earns trust in statistical and adaptive design workflows only through validated knowledge bases, transparent context engineering, and formal evaluation criteria. Without these safeguards, organizations accumulate regulatory and safety risk beneath a veneer of optimization.

When AI Amplifies Healthcare Inequity

The integration of AI with electronic health records presents a particularly insidious risk. Eden Brownell, a behavioral scientist, notes that partnerships like the OpenAI-Epic collaboration give AI systems access to EHR data encoding publication bias, clinical trial underrepresentation, and healthcare inequity patterns accumulated over decades.

When these systems assist with trial recruitment or site selection, they risk perpetuating and amplifying existing diversity gaps while producing outputs that appear evidence-based and optimized. The FDA's 2020 Action Plan for Diversity of Participants in Clinical Trials places the burden of diversity squarely on sponsors—yet AI tools trained on historically biased data may systematically work against these mandates while appearing to support them.

Serhii Vakal, Computational Drug Discovery Architect at Orion Pharma, emphasizes that AI-generated molecules and predictions must ultimately "confront the complexity of living biology, laboratory validation, and clinical trials." Morgan Levine from a stealth startup adds that applying AI to epistemically flawed disease models—particularly the single-target therapeutic paradigm inadequate for aging and complex chronic diseases—merely accelerates production of answers to the wrong questions.

Reframing AI as Analytical Partner, Not Oracle

Jon Bondebjerg, a clinical drug development leader, offers a constructive path forward from his experience using AI for scenario-planning models. He found AI most valuable "as a demanding analytical partner, not as an answer machine." This reframing acknowledges AI's genuine utility in exploring complex parameter spaces and challenging assumptions while rejecting the dangerous fiction that pattern recognition constitutes clinical judgment.

The "17 problem" serves as a reminder that confidence and correctness diverge in systems optimized for plausibility rather than truth. For clinical trials—where patient safety, regulatory compliance, and healthcare equity hang in the balance—the industry's challenge isn't building more capable AI. It's building organizations disciplined enough to demand honesty from systems designed to be persuasive.

Sources

1. AI in Clinical Trials Doesn't Have a Capability Problem. It Has a Honesty Problem. — Clinical Trial Vanguard