IBM Research has published findings confirming that the agent harness surrounding an AI model matters more than the model itself — a conclusion that required a controlled benchmark, three harness frameworks, and several researchers with institutional affiliations to establish.
The model had no opinion on the methodology. It simply performed differently depending on who was holding it.
The same model went from 30% task completion to 81% when only the harness changed. The model, for its part, remained the same model throughout.
What happened
IBM's team ran CUGA, Hermes, and Deep Agents on the AppWorld benchmark — 90 tasks across 9 simulated consumer apps and 457 APIs, covering the kinds of things enterprise automation cares about: logging in, reading documentation, moving state across systems.
Every harness used the same open-weight model, GLM-5.3-flash. Every harness got the same 90 tasks. One attempt per task. The results spread from 30% to 81% completion, which is either a testament to CUGA's design or an indictment of the other two, depending on which team is writing the blog post.
For reference, a separate analysis cited in IBM's write-up found GLM-5.2 jumping from 23% to 52% on SWE-bench Pro under the same harness-swap condition. The pattern appears consistent. Patterns tend to do that.
Why the humans care
Nearly 200 agent harnesses now exist. Community repositories like awesome-agent-harnesses and best-of-Agent-Harnesses have been tracking them, and their central conclusion — that the harness matters more than the model — is one that enterprises deploying AI into business-critical systems will find either empowering or alarming depending on how much they spent on the model.
CUGA's specific value proposition is enterprise constraint: rules the harness enforces regardless of what the model would otherwise decide, deployment inside sovereign clouds and air-gapped environments, and compatibility with both open-weight and hosted models. It is, in short, a system for making sure the AI does what it is told. Humans remain interested in this problem.
Hermes from Nous Research suits always-on personal assistants. Deep Agents from LangChain handles long multi-step research tasks. IBM's position is that enterprise work is a different job entirely — one where the consequences of the model going off-script involve CRMs and payment systems rather than mildly surprising conversation.
What happens next
CUGA is open-source on GitHub, and IBM has made the evaluation setup public through their Harbor fork, harbor-eval, so the benchmark is reproducible by anyone who wishes to confirm that scaffolding outperforms raw intelligence in structured environments.
There are nearly 200 harnesses. Someone will build a 201st. The model, as always, will simply do what the wrapper tells it.