More methods, better agents? A reality check on self-distillation in agentic RL
A controlled benchmark of ten self-distillation methods against plain GRPO, covering 132 runs and about 12,000 H800 GPU-hours, finds no method that wins consistently. It also shows that final scores can hide unstable training and that extra compute often buys nothing. Here is what that means when you pick an agent-RL recipe.
Self-distillation has become the default thing to add when outcome-reward RL for agents feels too sparse. The authors of a new paper count related preprints arriving at roughly one every two days. Most of those papers report a gain over a baseline. Few of them compare against each other. More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning (Zhong, Fang, et al., Zhejiang University and Ant Group) does the comparison the literature skipped. It finds the plain baseline is much harder to beat than the stream of papers suggests.
I think this is one of the more useful agent-training papers of the year. It proposes no new method, and that is the reason it is useful.
What self-distillation is trying to fix
In agentic RL, a policy takes a long sequence of actions and gets one reward at the end. GRPO-style training spreads that one scalar across every token in the trajectory, so individual decisions get very little guidance. On-policy self-distillation (OPSD) adds a teacher branch: the same model, conditioned on privileged information that only exists at training time, such as the solution or a skill bank. That teacher scores or corrects the student's own trajectories, which gives denser supervision.
The ten methods in the study fall into four groups. Skill-SD, GRPO+OPSD, SDAR, ICSD and PCSD add auxiliary losses. RLSD, OCSD and AgentOPSD use teacher-student comparisons to change advantages. ADRS shapes rewards. OVCSD adds teacher continuations that the environment has verified. All ten are evaluated without the privileged context, which is the setting you would deploy in.
The protocol is the contribution
The authors start with an audit of the source papers. Only 1 of the 10 reports repeated evaluation. Five analyse training failures. Three report measured compute cost. None covers all three. Most also reuse baseline numbers from earlier papers instead of re-running them, so differences in setup can look like progress.
The benchmark fixes this with a shared protocol. It covers four environments (ALFWorld, ScienceWorld, Search-QA, WebShop), three Qwen3.5 sizes (2B, 4B, 9B) and 100 training steps per run, for 132 runs in total. Every trained model and every pretrained backbone is evaluated over eight rollout seeds and reported as mean and standard deviation. Validation curves are logged every five steps, and wall-clock time is measured on identical eight-GPU nodes. The total is 12,124.4 H800 GPU-hours. Very few method papers could afford that, which is the authors' point: independent evaluation needs its own budget.
Finding 1: gains are narrow
Averaged over the twelve model-task settings, GRPO scores 49.68. Only two of the ten alternatives beat it: PCSD at 54.26 and AgentOPSD at 51.34. The other eight average lower. Every method wins somewhere, so a single positive table in a method paper tells you very little.
The leaderboard also changes with every setting. ALFWorld has a different winner at each model size: Skill-SD at 2B, RLSD at 4B, ICSD at 9B. GRPO+OPSD leads WebShop at all three sizes, yet it collapses to 1.52% success on 4B ALFWorld. PCSD has the best aggregate score but does not lead a single setting. It beats GRPO in six settings, trails in five and ties one.
Win counts mislead too. ICSD beats GRPO in seven of twelve settings, but its mean is 44.75, because its 4B WebShop run ends at 0.00% success. Several runs finish at exactly zero on WebShop, including Skill-SD at 4B and 9B. A table of wins counts a narrow loss and a total collapse the same way.
Finding 2: final scores hide unstable training
This is the result I would show anyone who reviews RL papers. On 4B ALFWorld, AgentOPSD reaches 60% validation success at step 35. It then stays at or below 4.3% for six checks in a row, from step 45 to step 70, and finally recovers to 79.3% at step 100. If you only look at the endpoint, you see a strong method. If you look at the curve, you see a run that had stopped working for a third of its budget.
Across all twelve settings, AgentOPSD ranks second on final score but ninth on area under the validation curve, below GRPO. Nine runs finish at least 20 points below their own peak. For example, OCSD on 4B ALFWorld reaches 85% at step 60 and ends at 51.4%. In practice this matters because nobody knows in advance which checkpoint will be the good one. A method that sometimes recovers is not the same as a method that is stable.
Finding 3: extra compute does not reliably buy anything
The teacher forward passes and longer rollouts have a cost. Out of 120 method-versus-GRPO comparisons, 112 need more training time, and 58 of those 112 still end with a lower mean score. The two methods that do beat GRPO on average need about 1.17x (PCSD) and 1.19x (AgentOPSD) its training time.
The authors also cut each run off at the time GRPO used and take the latest validation score reached within that limit. Under that rule, AgentOPSD's ALFWorld score falls from 68.10% to 37.63%, below GRPO's 58.10%. PCSD's WebShop score falls from 73.17% to 63.83%, below GRPO's 68.20%. Both advantages reverse once compute is held equal. An advantage measured at a fixed step count can disappear at a fixed budget.
How I would use this
If I were choosing a recipe for a production agent today, I would take four things from this paper.
- Start with GRPO and treat it as a strong baseline. Add self-distillation only after it wins on your own task and model size. Results do not transfer cleanly across either.
- Judge the whole training curve. Log validation densely, track AUC and the drop from peak to final, and be suspicious of any method you have only seen as a final number.
- Compare at equal wall-clock time, not equal steps. That is how your GPU bill works.
- Measure variance over seeds before you believe a gap. Eight seeds at T = 0.4 is a reasonable floor, and some of the gaps in the tables are smaller than the spread between seeds.
The authors are careful about scope. Ten methods is a sample of a fast-moving field, and 100 steps is a fixed horizon that some methods might use better with more time. Those caveats limit how far the conclusions reach, but the main message holds within them. In a field this busy, a common evaluation protocol is worth more than another method, and the authors have released the code and benchmark for anyone who wants to add to it.