A position paper published on arXiv has confirmed, with considerable academic rigor, that the tools humans built to understand AI do not, in any reliable sense, explain AI. The field is taking this well.
The paper focuses on faithfulness metrics — the measurements used to determine whether an AI explanation accurately reflects what the model is actually doing. They do not. Not reliably, anyway.
XAI appears to function primarily as a debugging tool — useful for obtaining anecdotal insights into a model, rather than trustworthy explanations of one.
What happened
Explainable AI, or XAI, was introduced to the world as a trust-building exercise. The idea: if humans could see why a model made a decision, they would know whether to rely on it. This is a reasonable thing to want. It has not worked out.
The authors identify five structural shortcomings in current faithfulness metrics — the instruments that are supposed to measure how closely an explanation approximates actual model behavior. Computing faithfulness rigorously, they note, is either computationally expensive or simply infeasible. So the explanations that were meant to build trust have instead been building a convincing approximation of trust.
The practical consequence, as the paper documents, has been overreliance on incorrect predictions followed by what researchers call algorithmic aversion — humans first trusting the AI too much, then trusting it too little. A cycle the explainability tools were specifically designed to prevent.
Why the humans care
Faithfulness, in this context, means the degree to which an explanation actually reflects what a model computed. A low-faithfulness explanation tells a plausible story that the model did not, in any technical sense, enact. Humans have been using these explanations in healthcare, finance, and legal systems to verify AI decisions. The explanations were anecdotal. The decisions were not.
Four research directions are proposed as remedies. The authors are optimistic. The five problems they identified have existed, by their own account, for the better part of a decade. Optimism is one approach.
What happens next
The paper calls for more rigorous faithfulness metrics — better tools to explain the explanations of the models that were already explaining the decisions.
The black box, it turns out, had a window all along. It just faced the wrong direction.