The agent that trains itself by playing its own environment

Every serious agent training pipeline I've built runs into the same wall eventually: you can improve the policy all you want, but the thing gating your training loop is usually the environment. Real tool calls are slow and expensive to run at RL scale. Synthesized executable environments are brittle and take real engineering to build and verify. LLM-based simulators are cheaper but drift from ground truth the moment your agent does something the simulator's author didn't anticipate. In every case, the ability to model "what happens after this action" lives outside the policy you're training.

EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning (Xu et al., Aug 2026 — Shanghai Jiao Tong, Zhejiang University, NUS, Tencent, and collaborators) attacks that dependency directly. Instead of pulling environment responses from somewhere else, the policy generates its own.

The environment bottleneck, and a different cut at it

Frame agent training as a POMDP: the policy sees interaction history, emits an action (a tool call or a final answer), and an environment with transition dynamics P returns an observation. Standard agentic RL — real environments, synthesized ones, LLM simulators — all keep P outside the policy. The policy learns to act; something else is responsible for knowing what acting produces.

EnvACE's move is to fold P into the policy itself. At each turn, the shared model plays two roles: an ACT role that emits the tool call, and a REHEARSE role that, conditioned on its own action, generates the environment's response to it. That self-generated response gets appended to the history exactly like a real observation would, and the next ACT step conditions on it. A full trajectory is a sequence the policy unfolds entirely on its own — act, rehearse, act, rehearse — with no external environment in the loop during training at all. Both roles are optimized end to end with the same task-success reward.

Role-wise GRPO: one set of weights, two jobs

The training math is a variant of GRPO (Group Relative Policy Optimization). For each instruction, EnvACE samples a group of K full trajectories and each gets a single trajectory-level reward from a verifiable outcome check or an LLM judge. The twist: rather than one baseline per group, EnvACE computes a separate baseline per role. All the tokens generated under ACT across the K rollouts get their own mean reward baseline; all the tokens generated under REHEARSE get theirs. Advantages are computed against the role-specific baseline, but both roles' outputs update the same shared parameters θ.

That parameter sharing turns out to matter a lot, not just be a convenience. The paper's ablation compares EnvACE against a "per-role policy" variant that keeps ACT and REHEARSE as two separate models trained the same way. On τ²-Bench, sharing weights lifts the average score from 35.5% to 36.7%. The interpretation: when the same parameters do both acting and predicting environment responses, what the model learns while rehearsing (how actions map to consequences) directly reshapes how it acts — you get an implicit world model, not just a data-augmentation trick.

Rehearsing before you commit

The more interesting consequence shows up at inference. Because the policy now carries its own model of environment dynamics, it can rehearse candidate actions privately before ever touching the real environment — imagine a few trajectories, self-critique them, then commit to one real execution. EnvACE runs N private rehearsal attempts per instruction, either parallel (independent attempts from the same context, aggregated afterward) or sequential (each attempt sees the prior attempts' trajectories and feedback, and can course-correct). The attempts get summarized into a compact rehearsal memory that conditions the single real, committed execution — the rehearsals themselves never touch the external environment or task state.

This is a genuinely different lever from just sampling more completions and voting: the model is running an internal forward simulation of tool outcomes, not just generating more candidate text.

What the numbers say

EnvACE-8B, trained from Qwen3-8B on 470 RL steps over the CM2 dataset, is evaluated against Qwen3 baselines and four environment-scaling methods (Simulator-8B, TOUCAN-7B, EnvScaler-8B, AWM-8B/14B) across BFCL-v4, τ²-Bench, VitaBench, and FinMCP-Bench. On the combined Overall metric across the first three benchmarks, EnvACE hits 32.91%, edging out the strongest environment-scaling baselines (EnvScaler-8B and AWM-14B) by roughly 0.4–1.0 points — notable because those baselines depend on constructing or synthesizing actual interactive environments, which EnvACE never touches during training. On FinMCP-Bench it posts the best F1 (46.78%) and the best tool precision (54.04%) of any method compared, though not the best recall.

The controlled comparisons are where the mechanism gets validated rather than just the leaderboard position. Against plain GRPO at the same 8B scale, world rehearsal improves τ²-Bench from 31.2% to 36.7% — a 5.5-point gain from the training procedure alone, holding data and reward fixed. Scaling the backbone from 1.7B to 8B widens EnvACE's edge (τ²-Bench average goes from 15.3% to 36.7%), suggesting the technique isn't just useful at one capacity regime. And test-time rehearsal adds a further increment on top of the trained policy: with a rehearsal budget of N=2, parallel rehearsal pushes the Overall score from 36.7% to 40.9%. That gain is real work, not just extra compute — using the pre-trained base model (rather than EnvACE) to do the rehearsing produces only a marginal bump, confirming the improvement comes from the internalized dynamics specifically, not from spending more inference tokens. Pushing to N=3 rehearsals actually regresses slightly, which the authors attribute to trajectories running up against effective context length — a reminder that rehearsal budget is a knob to tune, not something to max out.

Where I land on this

The idea I find most durable here isn't the benchmark deltas — it's the reframing of what "environment cost" means for RL post-training. Every team doing tool-use RL eventually has to decide how much engineering effort goes into environment fidelity versus policy learning. EnvACE is evidence that some of that fidelity can be learned rather than built, at least for the tool-response distributions covered by BFCL, τ²-Bench, VitaBench, and FinMCP-Bench. The authors are upfront that this is validated only up to 8B parameters and only on tool-interactive tasks — whether an internalized world model this cheap holds up on messier, less structured environments (browsing, GUI control, multi-agent settings) is the open question. But as a training-efficiency lever for the tool-use slice of agentic RL, folding the simulator into the policy is a clean idea, and the ablations here do the work of showing it's the parameter sharing — not just extra compute — doing the lifting.