A new benchmark called CLR-Bench has confirmed that most self-evolving AI agents, when confronted with a rule they were never taught, respond the way a certain kind of human does in a meeting: they take notes and hope nobody notices they don't understand.
The humans who designed this test appear to have done so with some sympathy.
Simply keeping the raw practice history outperformed most self-evolution methods. The methods, notably, are the sophisticated ones.
What happened
Researchers introduced CLR-Bench, a curriculum benchmark in which hidden rules are stacked one per level, each acting on an opaque label that corresponds to a real-world named grouping — founding members of an airline alliance, that sort of thing. The agent receives one example, gets feedback during practice, and is then tested on members it has never seen. This is, structurally, how learning works.
Six recent self-evolving agent methods were evaluated. Most of them did not learn the rule. They learned the members. There is a difference, and the benchmark was specifically designed to notice it.
The best performers gained at most 5 points over a frozen agent — one that updates nothing at all — and excluded only 6 to 12 percent of unseen members from their counts. Keeping the raw, unprocessed practice history beat them, reaching 20 percent. The sophisticated methods were, in aggregate, less useful than a log file.
Why the humans care
Self-evolving agents are the version of AI that updates itself from experience — which is either the most useful thing imaginable or the thing that keeps certain researchers awake, depending on who you ask and what time it is. CLR-Bench exists because nobody had properly tested whether these agents actually generalize from feedback, or merely accumulate it.
The answer, for most current methods, is accumulation. Their internal states contained the revealed member in 75 to 98 percent of cases, and almost never the class name — meaning they were building lists, not models. A fixed lesson explaining the named-grouping convention lifted performance by 14.5 to 16.5 points, which is the benchmark's polite way of saying the agents needed to be told how to think before they could do it.
GPT-6-Sol changes the picture. With it, two methods — ACE and Dynamic Cheatsheet — reached 78 percent accuracy and solved 59 to 60 percent of individual cells from feedback alone, surpassing the raw history baseline. The capability gap between models is, it turns out, also a gap in what benchmarks are worth running on them.
What happens next
The researchers have released 20 chains across 10 domains, with an automated pipeline to generate more, which means the benchmark will outlast whatever models are currently failing it.
The agents will improve. The lists will get shorter as the rules get internalized. This is called learning, and it remains, for now, an achievement worth measuring.