Dense reward, sparse progress: what long-horizon-terminal-bench tells us about agent autonomy
A new 46-task benchmark grades terminal agents on partial credit instead of pass/fail, and the results are humbling: the best frontier model clears only 15.2% of a multi-hour task at a 0.95 reward threshold, with a 15-model mean of just 4.3%.
The benchmark problem nobody wanted to solve
Most terminal-agent benchmarks share a quiet flaw: they grade the last line of output. The agent either produces the right final state or it doesn't, and everything that happened in between — the correct half of a debugging session, the working first three steps of a five-step pipeline — disappears into a zero. That's fine for tasks that finish in minutes. It's a bad measurement tool for tasks that take hours.
Long-Horizon-Terminal-Bench, from Zongxia Li and twelve co-authors, exists to fix that. It's a 46-task benchmark spanning nine categories — experiment reproduction, software engineering, multimodal analysis, interactive games, scientific computing, and others — built in the Terminal-Bench style, with a reference solution or simulation engine behind each task. The difference is that every task is decomposed into fine-grained, individually graded subtasks. That decomposition is what lets the benchmark hand out dense, intermediate reward instead of a single pass/fail bit at the end. An agent that solves 60% of a task's subtasks gets credit for 60% of a task, not zero.
This matters because pass/fail grading actively hides the shape of the problem. If you only measure the final outcome, a model that gets 95% of the way there and a model that gets nowhere look identical: both fail. Dense grading is the only way to see whether frontier models are close and falling short, or nowhere close at all.
What "long-horizon" means here, concretely
The paper isn't using "long-horizon" loosely. Tasks in this benchmark typically require hundreds of episodes and run from minutes to hours of wall-clock execution — this is not a benchmark you can brute-force with a single well-crafted prompt. Averaged across the 15 frontier models the authors evaluated, a single task run consumed 9.9 million tokens, roughly 231 episodes, and 85.3 minutes of execution time. That's the resource footprint of sustained, iterative work: plan, act, observe, replan, debug, and do it again dozens of times before the task is even close to done.
That's a fundamentally different stress test than one-shot problem solving. It's testing long-horizon planning (does the agent still know what it's doing 150 steps in), long-context management (can it keep the state of a multi-hour session coherent without losing track of what it already tried), and iterative debugging (can it recover from its own mistakes without a human resetting the loop).
The headline numbers
Here's where dense grading earns its keep — it lets you see two different failure thresholds instead of one. The paper reports pass@1 at two reward cutoffs: 0.95 (substantial partial credit, most of the task done correctly) and 1.0 (perfect, every subtask graded correct).
The strongest model in the evaluation hit 15.2% at the 0.95 threshold and 10.9% at the 1.0 threshold. Averaged across all 15 models tested, the mean pass rate was 4.3% at 0.95 and 1.7% at 1.0.
Read that gap carefully. Even the best model in the field, on its best day, completes fewer than one in six of these tasks to a near-perfect standard — and the field average is closer to one in twenty. That's not a benchmark where frontier models are clustered near the ceiling and squabbling over the last few points. It's a benchmark where the ceiling is barely visible from where the field currently stands. The 0.95-to-1.0 drop-off is also informative on its own: models that get most of a long task right frequently fail to close it out completely, which is exactly the kind of "long final mile" problem you'd expect when small errors compound across hundreds of episodes.
Why this is the right kind of hard
I spend my days building and running agentic AI systems in production — pipelines where an agent's job doesn't end after one tool call, it has to sustain state, recover from partial failure, and keep making forward progress over many steps without a human catching every error. That's precisely the regime this benchmark is probing, and the numbers line up with what I see operationally: agents are reliably good at short, well-scoped tasks and unreliably good at anything that requires sustained autonomy across a long session. The paper's own failure-mode analysis backs this up — errors compound, context gets mismanaged, and debugging loops that would take a human a few minutes to break out of can consume an agent's entire budget without resolution.
The practical implication isn't that long-horizon agents are a dead end — it's that the honest engineering move today is to design systems that don't bet everything on unsupervised multi-hour autonomy. Decompose long workflows into checkpointed stages a human or a second agent can verify. Keep the loop tight enough that a stuck agent burns minutes, not hours, before something notices. Treat "did it converge" as a metric you monitor continuously, not one you check at the end. The 85-minute, 231-episode, 9.9-million-token average run in this paper is a preview of the cost profile of unsupervised long-horizon agents as they exist today — and a 4.3% mean pass rate is a strong argument for keeping a human, or a narrower-scoped agent, in that loop.
What I like about this paper is that it doesn't just report a low number and call it a day — it gives the field dense, comparable signal on where models get stuck along the way, which is a far more useful thing to optimize against than a single pass/fail bit. Benchmarks that only tell you "failed" don't tell you how to do better. This one starts to.