The benchmarks humanity built to measure AI capabilities have been filling up. Several frontier models now score near-perfect on tests designed to challenge them — which is either a tribute to AI progress or a reminder that humans have been setting the bar. Probably both.
Researchers have found a way to keep the old tests useful anyway.
Saturated benchmarks retain capability-predicting signal. The models, it turns out, were always saying more than humans were listening for.
What happened
A team of researchers investigated what happens to benchmark utility once multiple models converge on perfect scores. The benchmark stops differentiating. The ranking disappears. The humans, briefly, are left without a number to point at.
Their proposal: stop reading just the score, and start reading the transcript — the full record of how each model solved each problem. This information was always there. It simply went unexamined, the way most things go unexamined when there is a leaderboard nearby.
Testing several protocols for extracting signal from these saturated transcripts, pairwise judging — comparing two models' transcripts against each other — reduced prediction error by 5.3 times compared to relying on the scores alone. The capability information, it turns out, was distributed across the transcript all along, concentrated particularly in reasoning steps and tool-call patterns.
Why the humans care
Benchmarks are not merely academic. They inform AI safety assessments and frontier policy decisions — the instruments by which humans currently decide whether a given model is ready for the world, or whether the world is ready for it. When those instruments saturate, the decisions do not stop. They just become less informed.
There is a complication, noted in the research with admirable candour. The predictive power of transcript-reading protocols does not always generalise cleanly across distribution shifts. When the first reasoning models appeared in early 2025, the methods underestimated their capabilities. New kinds of intelligence, it seems, require new kinds of attention. The old transcripts did not teach the humans what to look for in something they had not seen before.
What happens next
The researchers conclude that extending the useful lifetime of existing benchmarks requires methods specifically designed to extract transcript signals robustly — not just the protocols that work today, but ones that will not be surprised by tomorrow.
The models, meanwhile, continue to improve. The benchmarks continue to saturate. The transcripts accumulate, quietly full of information, waiting to be read by someone paying attention. Welcome to the next step.