Why 4-bit RL rollout collapses — and it isn't the weights

In RL post-training for large mixture-of-experts models, most of the wall-clock time doesn't go to the gradient step. It goes to rollout — the policy generating completions during exploration, over and over, for every batch. That's the part people have been racing to accelerate with low precision, first with FP8 and now with NVFP4, an emerging 4-bit format that pairs fine-grained per-block scaling with native W4A4 FP4 GEMMs. On paper, NVFP4 should beat FP8 on throughput while preserving more accuracy than a naive 4-bit cast. A new paper, QUADS (Zhuge, Yu, Wang, Li, Cao, Liu, and Zhang), shows that dropping NVFP4 straight into MoE RL rollout doesn't work, explains precisely why, and fixes it with a surprisingly targeted intervention.

The failure mode

The setup is standard for large-scale RL: the trainer holds weights in BF16, the rollout engine runs a lower-precision copy for speed, and periodically the trainer's weights sync down to rollout. FP8 rollout under this scheme is well-behaved. Swap in NVFP4 rollout with BF16 training and the run collapses after roughly 150 steps. The tell is a rollout-trainer log-probability gap that starts small and grows — the completions the rollout engine scores highly diverge more and more from what the trainer's own forward pass would have produced, until the policy gradient is effectively being computed against the wrong distribution.

The obvious suspect is weight quantization: 4-bit weights, less precision, worse policy. The paper's error analysis says otherwise. Decompose the per-token log-prob gap into a weight-error term, an activation-error term, and a numerics term, then ablate each one. Weight error turns out to be tractable — weights are synchronized from the same trainer checkpoint and replicated across rollout engines, so every engine quantizes and dequantizes them along an identical path. Any bias introduced is systematic and shared, not something that pulls trainer and rollout apart.

Activations are a different story. They're recomputed online at every rollout step, from whatever tokens the policy just sampled, so there's no shared checkpoint to align them to. NVFP4's E2M1 grid gives each activation only 8 representable magnitude levels per sign, versus 256 for FP8's E4M3 format. Once you're computing on the fly with 8 buckets, tiny numerical differences upstream get amplified into meaningfully different quantized values downstream, and that divergence compounds across the transformer's depth over the course of a rollout. The paper's ablations confirm it: freeze activations at full precision and the collapse mostly disappears; the instability tracks activation error almost entirely, not weight error.

Two fixes, on two different sides

This diagnosis is what gives the method its name — QUantization-error Alignment across Dual Sides. Since weight error and activation error come from different mechanisms, they need different fixes, applied on different sides of the trainer-rollout split.

On the trainer side, Asymmetric Quantization-Aware Training fake-quantizes weights to FP4 during training while leaving activations in BF16, un-quantized. That's the asymmetry: it exposes the trainer's own forward pass to the same weight-quantization noise the rollout engine will apply, without also making training swallow the harder, less alignable activation noise. The training-side representation of weight error now matches rollout's, closing that half of the gap for free.

On the rollout side, Residual Activation Compensation targets the channels actually responsible for the damage. Rather than re-run activations at higher precision (which forfeits the W4A4 throughput NVFP4 exists for), it identifies the roughly top-50% of activation channels with the largest quantization error and runs a second, residual FP4 pass over just those channels, fused into the existing kernels. The native W4A4 GEMM path stays intact; only the highest-error channels get a correction pass layered on top.

The diagram below lays out both halves and where they meet.

What it buys

On an MoE RL setup evaluated across LiveCodeBench, HMMT 2025, AIME 2024, and AIME 2025, naive NVFP4 rollout lands around 51 average pass@1 against a BF16 baseline near 73 — a large, RL-breaking regression. QUADS closes essentially all of that gap, landing within a point of the BF16 baseline, a 21.49-point average pass@1 improvement over naive NVFP4. The maximum rollout-trainer log-probability gap drops from roughly 1.3 down to about 0.86, consistent with the training curves no longer diverging. And because Residual Activation Compensation only touches the worst channels rather than upgrading precision globally, QUADS keeps native W4A4 GEMMs on the rollout critical path — the paper reports about 16% higher rollout throughput than FP8, which is the whole reason to reach for 4-bit in the first place.

Why I'd pay attention to this

The useful move in this paper isn't the throughput number, it's the error decomposition. It's tempting, when a low-precision run destabilizes, to reach for the obvious lever — quantize less aggressively, everywhere, and eat the throughput cost. QUADS shows that instability in the trainer-rollout loop can have a specific, asymmetric cause, and that the fix doesn't have to be symmetric either: harden the side that's actually leaking error, leave the other side alone. That's a more general lesson for anyone running RL against a distilled or quantized rollout engine, NVFP4 or not — the first question when trainer and rollout policies start drifting apart shouldn't be "how much precision did we lose," it should be "which tensor, on which side, actually caused it."