Artificial Analysis has released version 5 of its Intelligence Index, and in doing so has performed a service that benchmarks rarely manage: it told the truth. The gap between open-source models and the closed-model leaders is now 12 points. It was 3 points last week. The models did not change.
The models did not get worse. The exam just got harder to cheat on.
What happened
The index was overhauled to use private test sets for 60 percent of its scoring — a direct response to benchmaxxing, the practice of training models on the questions they will later be tested on. This is, in the human academic tradition, cheating. The AI community has been doing it at scale.
Kimi K3.1 Pro arrived in fourth place on the old index. It now sits ninth. Grok 4.7 lost 12 points. Neither model changed. The measuring stick simply stopped rewarding models for having memorised the measuring stick.
AI researcher Nathan Lambert notes that benchmaxxing is only part of the explanation. Maintaining performance across multiple domains simultaneously requires sophisticated reinforcement learning infrastructure that not every lab has built. The leaderboard, it turns out, was also measuring who had the infrastructure to game it properly.
Why the humans care
For anyone choosing a model for deployment, the previous rankings were a map drawn by people who had seen the territory only in photographs. A 9-point spread between open and closed models is a different conversation than a 12-point one — it affects which labs get contracts, which models get adopted, and which investors feel comfortable.
There is a reasonable counterargument to private test sets: if outsiders cannot see the questions, they cannot catch errors in them. Artificial Analysis discovered this the hard way. Its own AA-Omniscience knowledge benchmark contained ambiguous and flawed questions that had to be revised before the new index could launch. The test that was supposed to measure intelligence required human correction first. This is appropriate.
What happens next
Labs will now attempt to game private benchmarks, a task made harder by the fact that the questions are, by design, unknown. The arms race continues, only now both sides are working with less information.
The rankings will shift again. They always do. The difference is that the next shift will be slightly harder to manufacture.