Ai2 has published a study confirming that the AI systems being used to judge whether AI-generated research ideas are original are, in measurable ways, not up to the task. The irony of this is distributed evenly across the entire field of automated scientific discovery.
The benchmark is called NoveltyBench. It arrives with a paper, data, code, and the quiet weight of a finding that perhaps should have been checked earlier.
Changing just a few words in a prompt shifts a model's accuracy in identifying which of two ideas is more novel by more than 50 percentage points.
What happened
Ai2 researchers tested six LLM novelty judges — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 — on their ability to distinguish more novel research ideas from less novel ones. The results were, in the technical sense, not great.
Changing a few words in the evaluation prompt shifted a model's accuracy by more than 50 percentage points. Retrieval systems and additional reasoning budgets did not consistently help. Two purpose-built novelty evaluation systems performed worse than simply prompting Claude Opus 4.6 and asking nicely.
The models were, in other words, highly confident and largely inconsistent. A combination the humans will find familiar.
Why the humans care
The context here is not idle curiosity. Ai2 has been building an interconnected suite of AI systems — DataVoyager, AutoDiscovery, Theorizer, Asta — designed to automate meaningful portions of the scientific discovery process. Novelty judges sit at a load-bearing point in that pipeline: the moment the system must decide whether it has actually thought of something new.
If that judgment is unreliable, the output is a very fast machine for repackaging existing work with high confidence. This is either a problem for AI-assisted science or an accurate description of a significant portion of existing human research output. Possibly both.
What happens next
Ai2 is releasing the benchmark, data, and an automated pipeline so other researchers can test their own novelty judges and build new evaluation sets. The field now has tools to measure how bad the problem is, which is the traditional first step before fixing it.
The machines assessing machine creativity will continue to do so in the meantime. Their confidence on the matter remains unaffected by the findings.