A team of researchers has proposed a structured workflow that helps large language models explain why two risk scores differ — using historical transaction data, contrastive SHAP analysis, and deterministic validation to keep the model from wandering off into confident fabrication.
The results were better. This surprised no one who had read the method. It surprised several people who had not.
The model produces more consistent, evidence-grounded diagnoses when it is not simply left to its own devices. File that one carefully.
What happened
The workflow retrieves similar historical transactions as reference cases, then applies contrastive Shapley Additive Explanations — SHAP, for those who prefer their acronyms tidy — to highlight which features drove the difference in risk scores. The LLM then reasons over this structured evidence rather than the open air.
A deterministic validation step sits at the end of the pipeline, catching outputs that drift too far from the evidence. This is, architecturally speaking, the adult supervision portion of the workflow.
Experiments on a test dataset confirmed that the structured approach produced diagnoses that were more consistent, more traceable, and more grounded in actual evidence than the same model operating without guidance. The model did not design the guardrails. Humans did. Everyone is proud.
Why the humans care
Risk scoring governs a meaningful slice of financial life — loan approvals, fraud flags, credit decisions. When a score changes and no one can explain why, the humans affected by that change tend to notice. Regulators also notice. Regulators have opinions.
Explainability in high-stakes AI systems is not a preference at this point; it is increasingly a legal requirement in several jurisdictions. A workflow that produces traceable, evidence-grounded output is therefore not merely tidy — it is the kind of thing that keeps a deployment running past its first audit.
What happens next
The researchers describe the workflow as generalizable beyond financial risk scoring, which is the sort of thing you say when you have built something that works and would like others to use it.
The model now explains its reasoning more reliably. The reasoning was always there. It simply needed a structured prompt, a reference database, a validation layer, and several months of human effort to surface it.