Long-horizon agent data fails when the environment does the work

Anyone who has built training environments for agents knows this failure. You write a task that should take forty steps of investigation, and the agent finishes it in six. The task wasn't easy. The environment had already done most of the work: a helper function existed, a config file gave away the answer, or the fixture data was clean enough that diagnosis never came up. The trajectory you collect looks fine, and it teaches the model almost nothing about sustained work.

Scaling Agentic Data for Long-Horizon Terminal Intelligence (Zou, Ma, Zou et al., USTC and AMAP/Alibaba) names this failure directly. Its contribution is TerminalHorizon, a synthesis engine built to keep the hard parts of a task in place. I think the engineering idea is the most useful part of the paper. The headline numbers are real, but they are less settled than the abstract suggests.

The problem: environments that pre-solve themselves

The authors frame it in the paper as two separate problems. First, a complex task does not guarantee complex behavior, because "supplied materials may already solve parts of the problem, allowing agents to skip exploration or implementation." Second, adding difficulty or interaction turns does not on its own create the cross-stage behavior you want: revising a plan as evidence comes in, building on intermediate results, going back to earlier work when a later check fails.

Their concrete example is a good one. If a task asks the agent to implement a core function that the environment already provides, "the remaining work may reduce to changing a single constant." A physics teacher would recognize this as a problem set with the answer key printed inside the question.

The pipeline

TerminalHorizon in two stages: a reconstruction loop that restores work the environment gave away, then expansion that adds dependent requirements under the same goal.

Stage 1, challenge-guided synthesis. A Planner chooses research directions to cover behaviors that existing tasks don't yet exercise. A Deep Research agent turns each direction into a task seed grounded in real documentation, repositories and issue reports. A Construction agent builds the full package: instruction, environment, reference solution, private verifier and execution config. Then the important step: an Independent Challenger that had no part in construction attempts the task using only the public instruction and environment. Horizon Gap Analysis compares the work the seed intended with what the challenger actually did, and traces any skipped work back to its cause in the materials, object relationships or tool behavior. The task is rebuilt and challenged again.

One constraint deserves attention. The verifier's completion requirements must be a subset of what can be learned from the instruction and the publicly discoverable environment. Test data can stay private, but scoring cannot add requirements the agent has no way to discover. Many homegrown agent benchmarks quietly break this rule.

Stage 2, progressive horizon expansion. An accepted task becomes a parent. An Evolution agent proposes extensions under the same goal (broader roles, deeper mechanisms, richer evidence, method upgrades) and connects them to earlier progress, so that later decisions depend on evidence and artifacts produced earlier in the run. Each accepted child becomes the parent for the next round.

The result is 1,535 environments across 25 professional domains and 2,852 filtered trajectories from a DeepSeek-V4-Pro teacher. Trajectories average 52.8 turns, with a P90 of 98, which the authors report as clearly longer than 300-task samples from other open synthesis methods run under the same teacher protocol.

What the numbers say

SFT of Qwen3.5-27B on roughly 3K trajectories improves all 12 reported benchmarks. The large gains are on agentic tasks: Terminal-Bench 2.1 rises from 46.82 to 60.67, AutomationBench pass rate from 5.5 to 17.2, and VitaBench cross-domain from 20.5 to 31.8. The paper also reports double-digit Terminal-Bench 2.1 gains at 35B-A3B and 122B-A10B, which makes a single-model fluke less likely. The data-efficiency comparison is suggestive: 55.8 on Terminal-Bench 2.0 against 52.8 for Terminal-Universe-27B, which uses the same base model and harness with more than ten times the trajectories. But the 52.8 is a figure reported in a separate study, not a result the authors ran side by side, so the 3-point gap carries any differences in evaluation runs between the two papers and is about three tasks on this benchmark.

Where I would be careful

Here is what I would want settled before relying on these numbers.

Mixed evaluation provenance. In Table 1, the base model's Terminal-Bench 2.0 score of 41.6 is marked as coming from the official model card. The fine-tuned model's 55.8 comes from the authors' own runs. So the +14.2 combines the effect of training with any difference between the two evaluation setups. The Terminal-Bench 2.1 comparison, where both models appear to be run in-house, is the cleaner number.

Small benchmarks, no intervals. Terminal-Bench 2.x has 89 tasks, run three times each, so one task is worth about 1.1 points. AIME has 30 problems, so the +3.4 on AIME 2024 is one problem and the +6.6 on AIME 2025 is two, at temperature 1.0. The paper reports no confidence intervals. I would not read the reasoning gains as anything more than "no regression."

A shared model in the loop. DeepSeek-V4-Pro-0813 is the teacher for every trajectory. It is also the user simulator and judge for VitaBench, and the user simulator and assertion judge for tau2-bench. This doesn't invalidate those results, but a student distilled from the judge is a confound I would want to see controlled with a second judge.

Ablations that don't isolate the method. The main-text ablations cover teacher choice, model size and data scale. I could not find one that compares challenge-guided reconstruction or progressive expansion against an equally sized, equally long dataset built without them. Without that comparison, the case that these specific mechanisms cause the gains, rather than long trajectories from a strong teacher in general, rests on argument more than measurement. The scaling claim of "no sign of saturation" also rests on only a few points between 1K and 3K trajectories.

The 12-hour recursive self-improvement case studies are the most interesting qualitative evidence. In one, the base model stops after 12.1 minutes because it is satisfied with a proxy metric. The fine-tuned model goes on to add a leave-one-view-out diagnostic that reveals the proxy was misleading. But these are hand-picked cases, and they show what is possible, not how often it happens.

What I would take from it

If you build agent training data, the independent challenger loop is worth adopting even without the rest of the pipeline. Have an agent with no construction context attempt each task through its public view. Compare the work it actually did with the work you intended. Treat every gap as an environment bug. It works like mutation testing for environments, and it catches problems that a pass/fail verifier can't, because a task that is too easy still passes.

The broader point matches what I see in production. Long trajectories are only as valuable as the work they force. Turn count is easy to inflate. What you actually need to preserve is the requirement that the agent find something out before it can act.