How Kaiser Permanente Built an AI Scorecard to Predict Which Tools Will Actually Work at Scale

As healthcare systems race to deploy artificial intelligence, Southern California Permanente Medical Group (SCPMG) has developed a governance framework designed to predict which AI tools will scale beyond pilot projects.

The Collaborative Technology Adoption for Care Transformation (C-TACT) framework, published in npj Digital Medicine, represents a shift from traditional risk-focused oversight to operational scorecards that evaluate AI across its entire lifecycle—from vendor selection through long-term maintenance.

From Risk Mitigation to Operational Reality

Traditional AI governance frameworks concentrate on pre-deployment risk assessment, but SCPMG's approach incorporates business viability and operational execution as weighted domains throughout the AI lifecycle. The framework organizes governance across three pillars—People, Process, and Technology—and four distinct lifecycle stages: Vendor Evaluation, Pilot, Scale, and Maintenance.

What distinguishes C-TACT from existing models is its explicit inclusion of Scale and Maintenance phases, which incorporate business and operational execution metrics alongside technical performance. This addresses a critical gap: many AI tools succeed in pilots but fail to achieve system-wide adoption.

The framework employs domain-weighted scorecards using Analytic Hierarchy Process methodology, achieving a consistency ratio of 0.0037 for weighting domains. This mathematical approach provides systematic prioritization across competing governance considerations.

Retrospective Testing Shows 30-Point Separation

When applied retrospectively, the Pilot Score Card separated two published SCPMG AI deployments from four anonymized comparators by approximately 30 points, with scores ranging from 47.0 to 53.5 across three different weighting schemes.

However, the authors explicitly state this represents a methodological demonstration rather than validated discriminative performance. They emphasize that prospective, blinded validation is needed to establish whether the framework reliably predicts successful AI deployment at scale.

The framework was developed by SCPMG's AI Advisory Council, which includes clinical, operational, financial, and technical expertise. This multidisciplinary approach reflects the reality that successful AI deployment requires alignment across domains that traditionally operate in silos.

Implications for Healthcare Systems and Vendors

For healthcare providers, C-TACT offers a systematic approach to evaluate AI tools beyond initial technical specifications, incorporating the operational and business factors that determine whether a tool will deliver sustained value in real-world settings.

For health technology companies, understanding this framework may become critical for successful adoption in value-based care systems. The weighted scoring across lifecycle stages represents how a major integrated delivery network evaluates which AI solutions merit investment and scale.

The framework's emphasis on operational learning and maintenance phases acknowledges that AI deployment is not a one-time implementation but an ongoing process requiring sustained governance and adaptation. This perspective aligns with the realities of value-based healthcare systems, where long-term operational efficiency and clinical outcomes determine success.

SCPMG has trademarked "C-TACT" and the full framework name, suggesting potential broader dissemination. No specific funding was received for this work, indicating development emerged from operational necessity rather than external research support.

As healthcare AI deployment accelerates, governance frameworks that match the pace of innovation while maintaining rigorous oversight will become essential infrastructure for health systems navigating the transition from pilot projects to scaled implementation.

Sources

1. Clinical AI governance for scalable innovation in value based healthcare systems — npj Digital Medicine