Why group sampling breaks asynchronous agentic RL, and what GLM-5.2 used instead

Reinforcement learning for LLM post-training has mostly run in lockstep: generate a batch of rollouts, wait for all of them, update the policy, repeat. That works fine when episodes are short and roughly the same length. It falls apart for agentic tasks — multi-turn coding sessions, long tool-use chains — where one trajectory can take ten times longer than another. Anyone building asynchronous RL infrastructure eventually runs into the same question: what do we do about GRPO's group-wise sampling?

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning, from Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong, answers it directly. The method — SAO — isn't a new capability or a benchmark win dressed up as research. It's an infrastructure and stability fix for a well-known mismatch, and it shipped: the paper reports SAO was used to train the production GLM-5.2 model (750B total parameters, 40B active per token, mixture-of-experts). That production detail is what made me read past the abstract.

Why GRPO doesn't fit asynchronous training

GRPO's group-wise sampling generates several rollouts per prompt, estimates advantage relative to the group mean, then trains. In an asynchronous rollout system, the grouping is exactly where things break: as the authors put it, "the group has to wait for the slower one to finish before fed into training." In long-horizon agentic settings — SWE-bench-style multi-turn coding, tool calling with retries — trajectory length is unpredictable, so the whole group is bottlenecked by its slowest member. Worse, plenty of realistic online-learning setups only ever hand you one trajectory per prompt in the first place, which makes group-wise advantage estimation a non-starter rather than just a slowdown.

Single rollout, direct importance correction

SAO's fix is to drop the group entirely: one rollout per prompt, fed into training the moment it finishes. That removes the straggler problem, but it reintroduces an older one — off-policy drift. By the time a trajectory completes, the policy has usually moved past whatever version generated it, sometimes across several intervening updates within a single rollout.

The textbook fix is importance sampling against the exact rollout-time policy, π_θ_old. But in an asynchronous system where the rollout engine's weights update mid-generation, tracking that exact behavior policy is, in the authors' words, "computationally prohibitive." SAO's answer is to stop pretending it's exact: use π_rollout — the actual weights the rollout engine was running — as the behavior-policy proxy directly, computing the importance ratio r_t(θ) = π_θ / π_rollout and dropping π_θ_old from the picture entirely.

Double-sided clipping, not just clipping

PPO-style clipping bounds the objective, but gradients can still leak through outlier tokens with extreme ratios. SAO applies what it calls a strict double-sided token-level clip: any token whose ratio falls outside [1 − ε_low, 1 + ε_high] is masked out of the gradient computation entirely, rather than clamped to the boundary. In their reasoning setup that's ε_low = 0.3, ε_high = 5.0; for coding it's ε_low = 0.8, ε_high = 3.0 — asymmetric bounds that tolerate more upside than downside drift, and simply discard the extreme cases instead of trying to correct for them.

Fixing the value model, not just the policy

The less obvious contribution is what SAO does to the critic. On inspection, the value model's gradient norms turned out to be far more volatile than the policy's, and the instability traced specifically to the full-attention layers rather than the mixture-of-experts layers. SAO freezes the pretrained attention weights in the value network and trains only the MoE projections, on the reasoning that attention already knows how to attend to the right tokens — what needs updating is the value estimate itself. Alongside that, they run two value-network updates for every one policy update (K=2), and use a "skip-observation" variant of GAE that bridges advantage estimation across environment-generated tokens the model never produced, instead of treating them as ordinary transitions.

What the numbers show

On Qwen3-30B-A3B, SAO beats both a GRPO-plus-importance-sampling baseline and the raw baseline on every benchmark reported: AIME2025 (97.3% vs. 93.5% vs. 85.0%), BeyondAIME (74.8 vs. 70.8 vs. 63.0), HMMT Nov 2025 (88.3 vs. 84.0 vs. 76.7), and IMOAnswerBench (74.0 vs. 70.0 vs. 55.3). On SWE-Bench Verified, SAO reaches 29.8% against 27.0% for GRPO and a 23.0% baseline. The more telling result is stability: standard GRPO collapses around step 160 in their asynchronous setup, while SAO trains stably out to 1,000 steps.

The authors also ran a simulated online-learning test where the reward criterion itself shifts mid-training — cycling a style preference between "cute," "chuunibyou," and "classical" response tones. SAO's critic tracks the moving target and recovers quickly after each shift; a running-mean baseline lags noticeably and settles at a lower ceiling. It's a synthetic test, but it's the right one for a method whose selling point is behaving well when the environment underneath it keeps changing.

Why this is worth reading

There's no new capability here, and the authors don't claim one. What SAO offers is a defensible way to run agentic RL without synchronizing on stragglers, using a value model that doesn't fall over halfway through training. That's the unglamorous part of scaling RL post-training — the part that shows up in stalled runs and outage reports, not in leaderboard screenshots. The fact that it's already load-bearing for GLM-5.2 in production is a stronger signal than another point of benchmark accuracy would have been: it means the stability holds outside a controlled ablation, at the scale where async infrastructure actually gets tested.