Can LLMs build and evolve their own agent harness? A new benchmark checks

The infrastructure question nobody was measuring

Most agent benchmarks hand a model a fixed harness — the tool definitions, retry logic, prompt scaffolding, and orchestration code that sits between a raw model and a working coding or research agent — and then measure how well the model performs inside it. That's a reasonable thing to measure, but it quietly assumes the harness itself is a solved, external variable. HarnessDev, a new benchmark from Yuhao Wu and eighteen co-authors, asks a different question: can a model build that harness itself, and can it make its own harness better over time?

This matters more than it sounds. I've watched teams swap out an agent's tool-calling loop, retry policy, or context-management scheme and see downstream task performance move by double digits, with the underlying model weights untouched. The harness isn't packaging around the model — it's load-bearing. If self-improving agents are going to be more than a slide in a keynote, one of the things they need to self-improve is the scaffolding they run inside.

Two stages: build it, then evolve it

HarnessDev splits the problem into two stages.

Creation. The agent starts from a minimal seed and a small number of example cases, and has to construct a complete, runnable execution system from scratch — not fill in a template, build the thing.

Evolution. Starting from the harness it just created, the agent iterates on its own infrastructure using downstream execution feedback, trying to raise its own benchmark performance across repeated revision cycles.

Every harness that comes out of either stage gets scored on two separate axes: capability (task success on held-out benchmarks the agent never saw during development) and efficiency (execution-token cost — how expensive the harness is to run, independent of whether it works). Separating those two is the right call. A harness that scrapes together a few extra correct answers by burning three times the tokens on retries isn't obviously better than a leaner one that's slightly less accurate, and production teams have to weigh both. This benchmark refuses to collapse them into one number.

The scale of the test

The Creation results span six creator LLMs, four task domains, and five downstream benchmarks, totaling 2,207 unique downstream instances. Critically, the evaluation tasks used to score the final harnesses are hidden — withheld from the model during development, so a harness can't be quietly overfit to the test it will be graded on. That detail is what makes the results worth taking seriously rather than dismissing as another leaderboard.

A split verdict, by domain

The headline finding isn't uniform failure or uniform success — it's a domain split. On code, and on search and research, the harnesses these models built stayed substantially behind mature, human-engineered reference harnesses. These are exactly the domains where the reference implementations have accumulated years of accreted judgment: how to sandbox execution safely, when to retry a failed test versus abandon the approach, how to paginate and deduplicate search results, when a citation is trustworthy enough to stop digging. That's the kind of engineering taste that doesn't show up cleanly from a handful of seed examples.

On writing and machine-learning experimentation, though, model-built harnesses matched or exceeded the selected references. Both domains reward something closer to a well-structured prompting and iteration loop than deep systems engineering: draft, critique, revise; set up an experiment, run it, read the result, adjust. That's closer to what these models already do well, and it shows.

Execution cost also varied widely across the generated harnesses, which is its own warning. Two harnesses that land on similar capability numbers can differ sharply in how many tokens they burn getting there — a gap you'd never see if you only reported the capability column.

Evolution's gains don't travel

The Evolution stage is where the paper does the most damage to the self-improving-agent narrative. Agents iterating on their own harnesses did produce performance gains — but the gains were unstable across revision runs, and transferred only partially to held-out tasks. A harness that improved on the tasks it was iterating against didn't reliably carry that improvement to problems it hadn't seen, which is precisely the property you'd want before trusting this process unsupervised.

The paper pushes further with a fixed-runtime-model experiment: hold the model that executes the harness constant, and vary what built it. The gains from evolution turned out to depend strongly on which model was actually running the harness — meaning improvements tuned around one model's quirks don't transfer cleanly to another. A harness one model evolved to suit its own habits isn't a harness another model can simply inherit.

What I take from this

I read this as a useful corrective, not a discouraging result. The domain split is the actionable part: if you're building agentic systems for code or research-heavy retrieval, don't expect a model to bootstrap harness quality anywhere close to what a deliberately engineered system achieves — that gap is where human infrastructure work still earns its keep. For writing and experimentation-style workflows, model-generated scaffolding is already competitive, and it's reasonable to let an agent draft its own loop there.

The evolution results are the more important caution for anyone imagining agents that quietly get better at their own execution infrastructure over time with no supervision. Right now, that improvement is real but fragile — it doesn't reliably generalize to new tasks, and it doesn't transfer across models. Self-evolving harnesses aren't a hype claim to dismiss outright, but on this evidence they're an early-stage capability, not an infrastructure strategy to build a production system around yet.