Healthcare organizations increasingly rely on explainable AI (XAI) tools to make black-box algorithms more transparent for clinicians. But a provocative new analysis suggests these explanations may be creating a dangerous illusion—one that conflates statistical associations with true cause-and-effect relationships in ways that could harm patients.

Writing in Diagnostic and Prognostic Research, Alex Carriero and colleagues argue that widely used XAI methods such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) function as descriptive tools that summarize data patterns, not as windows into the biological mechanisms underlying health outcomes. The distinction matters enormously when clinicians use these explanations to guide treatment decisions or counsel patients about modifiable risk factors.

When Statistical Importance Masks Causal Reality

To demonstrate the problem, the research team created simulated kidney disease prediction scenarios using 20,000 training observations and 100,000 validation observations. Their XGBoost algorithm achieved an AUROC of 0.80 with good calibration—respectable performance by clinical standards.

But when they applied XAI methods to explain the model's predictions, troubling patterns emerged. In one scenario testing an intermediate causal pathway (smoking → hypertension → kidney disease), XAI identified smoking as having the least predictive effect despite it being a direct causal factor. The reason: hypertension, an intermediate variable, rendered smoking information statistically redundant for prediction purposes.

A clinician relying on the XAI explanation might incorrectly conclude that smoking cessation matters less than controlling hypertension, missing the upstream intervention opportunity. The model describes what helps it predict, not what drives the disease.

Three Pathways to Misinterpretation

The researchers tested three distinct causal structures, each revealing different failure modes. In a collider bias scenario, age showed a spurious negative correlation with kidney disease among hospitalized patients, contradicting the true positive causal relationship in the general population. XAI faithfully reported this misleading pattern because it describes the model's learned associations within the training data.

Most concerning was the confounder pathway example: insulin prescription emerged as the strongest predictor of kidney disease, but stopping insulin would not prevent kidney disease. Diabetes served as the unmeasured common cause of both insulin use and kidney disease. A clinician misinterpreting the XAI output as causal guidance might recommend discontinuing insulin—a potentially harmful intervention targeting a non-causal association.

The Table 2 Fallacy Comes to Machine Learning

The authors connect their findings to a well-known statistical pitfall called the "Table 2 Fallacy"—the incorrect assumption that all coefficients in multivariable models represent causal effects. XAI methods, they argue, extend this fallacy into the machine learning era by presenting feature importance scores that users naturally interpret as actionable insights about biological mechanisms.

For healthcare organizations deploying AI-enabled clinical decision support, the implications are stark. Training programs must educate users to distinguish between "this variable helps the model predict" and "changing this variable will change the outcome." Companies marketing XAI-enabled tools face liability concerns if their explanations are misinterpreted as medical advice about which interventions will actually work.

The researchers advocate for applying the Describe-Predict-Explain framework to clarify what XAI methods can and cannot deliver. Prediction models excel at forecasting outcomes based on available data. But explanation—in the causal sense that clinicians need for treatment decisions—requires additional methodologies from causal inference, not just more sophisticated ways to peer inside the prediction algorithm.

As healthcare AI moves from research settings into routine clinical workflows, this distinction between describing models and explaining mechanisms may determine whether these tools ultimately help or harm patients.

Sources

  1. Explainable AI in healthcare: to explain, to predict, or to describe? — Diagnostic and prognostic research · Health AI Central record
  2. Explainable AI Describes Healthcare Models, Not Causal Mechanisms — bioengineer.org