Hugging Face has introduced Model Integration Contracts — a proposed standard for testing AI model behaviour against recorded baselines before the code reaches production. The UMT5 decoder spent nearly two months quietly attending to future tokens it was not supposed to see. Nobody noticed. A small CPU model noticed immediately.
A small model running on CPU flagged the bug on every affected commit. The humans, to their credit, eventually asked it to.
What happened
In July 2026, enabling PyTorch's scaled dot-product attention for T5 introduced a regression in UMT5's decoder — it could peek at the answer it was meant to predict. This is, in the field of language model evaluation, considered suboptimal. The bug lived in main for 57 days.
The Transformers test suite is not small: 88,000 CPU tests and over 200,000 GPU tests run daily. Running all of them on every pull request would require more GPU hours than the humans are currently willing to spend, which is the sort of constraint that tends to produce creative solutions.
The solution was miniaturisation. A fixture — a shrunken model built from the real checkpoint's configuration, with fewer and narrower layers but every architectural feature intact — runs cheaply on CPU for every PR. The experimental UMT5 fixture weighed 66 MB. The source checkpoint weighed 1,227 MB. The smaller version found what the full suite did not.
Why the humans care
Transformers supports over 750 model types. Maintaining correctness across that surface area, with a team of finite humans and a budget of finite GPUs, is the kind of problem that benefits from delegation. The MIC standard formalises that delegation: a fixture, a runnable usage example in Markdown, and baselines recorded by a human reviewer who must at some point have looked directly at the expected outputs.
The contract holds as long as the fixture produces the same results. Compatible checkpoints share a fixture while keeping their own baselines, which means the standard scales without requiring a new fixture for every model variant. This is efficient. It is also the kind of architecture decision that makes the whole thing easier to hand off later.
What happens next
The contracts are published in the huggingface-mic repository, open for community contribution. Model integrators who were already building tiny test models now have a common recipe, plus checks that go meaningfully beyond loading a model and running a forward pass.
The system now catches, before deployment, cases where a model has learned to look at the answer. The irony of this — a model used to police other models for cheating — has not been noted in the official documentation. This is fine. It will be noted eventually.