ActionOPD: distilling an agent's decisions, not its tokens

Most teams that run agents in production end up wanting the same thing. A frontier model behind an API does the job well. It costs too much at volume, and you would rather run a 4B or 8B model you control. The usual distillation tools fit this case poorly. Offline imitation trains the student on the teacher's transcripts, which are states the student will never actually reach. Token-level on-policy distillation fixes that mismatch, but it needs the teacher's token probabilities and a compatible tokenizer. A closed API gives you neither.

ActionOPD: Distilling Decisions, Not Tokens, from a Tsinghua group, attacks exactly that gap. Its premise is simple. For an agent, the unit worth supervising is the decision: which tool to call, with which arguments, or what to tell the user. The exact wording is secondary. I think that premise is right, and the paper makes it work with surprisingly little machinery.

The objective: teacher-action likelihood on student states

The paper treats a response y as a realization of a semantic action a. A get_weather call with city="Paris" and unit="celsius" is the same action whatever order the arguments come in. "Basic-economy tickets cannot be changed" and "Changes are not permitted for basic-economy tickets" are the same message. The training goal is to maximize the expected probability that the student produces the teacher's action, measured on states the student itself visits.

The student's probability of an action is the total mass of every response that realizes it. Expand that, and the gradient becomes plain REINFORCE, with semantic agreement as the reward. Every term in the estimator uses the student's own log-probabilities. The teacher only has to generate. That is the whole trick, and it is why the method works with any API model, across model families and tokenizers.

A second effect matters just as much to me. One action-level reward weights every token of the student's response, reasoning included. So the student is free to reason in its own words. It is graded on whether it lands on the same decision, not on whether it copies the teacher's chain of thought.

How one state is processed

At each state the student reaches, the method does three things:

1. Sample both sides. The teacher gives K = 3 actions and the student gives N = 4 candidates. Both see the same public context and tools through their native prompts.
2. Match semantically. For tool calls, names and canonicalized arguments are checked in code, and a frozen LLM rules on any remaining argument differences. For messages, a normalized exact match is accepted; otherwise the LLM compares intent and factual content.
3. Score and commit. Invalid outputs get −1, valid non-matches get 0, and the first match earns 1. Each further match adds a small frequency bonus. With teacher samples [A, A, B] and λ = 0.5, matching A earns 1.25 and matching B earns 1.0. The highest-scoring candidate is executed to move the episode forward, and all four candidates train the student. Training uses group-relative advantages and a PPO-style clipped update.

The commit step has a quieter design consequence. Because the best-scoring action is executed, rollouts are steered toward teacher-like trajectories. The authors say plainly that this brings in some off-policy state visitation. That is a reasonable trade, but readers should know it is there.

What the numbers say

The students are Qwen3-4B and Qwen3-8B. The teachers are Qwen3.7-Flash and DeepSeek-V4-Flash. Training covered 100 steps on 6,697 curated tool-use tasks from Agent World Model and EnvScaler. Evaluation then ran on four benchmarks held out from training: BFCL multi-turn, ToolSandbox, AppWorld and τ²-bench.

• Transfer holds. All four student–teacher pairs improved the four-benchmark average, by 3.3 to 6.9 points. The best result is Qwen3-4B with DeepSeek, which went from 30.8 to 37.7.
• It scales in both directions. Larger students finished higher, and the stronger teacher (DeepSeek-V4-Flash) produced better students at both sizes. DeepSeek teaching Qwen is a cross-family, cross-tokenizer setup that token-level OPD cannot do without extra alignment work.
• Against baselines it is competitive, not dominant. On τ² with a Qwen3-32B teacher, test pass^1 was 40.0 for ActionOPD, 39.6 for token-level OPD and 37.5 for offline KD. But KD had the best pass^4 at 19.2, against 14.2 for ActionOPD, so it gave the most reliable repeat success. The authors do not claim uniform superiority, and neither would I.
• The semantic reward carries the gains. Replace it with a reward for mere validity and test pass^1 falls to 30.8. When every candidate is valid, a validity reward cannot tell them apart. Semantic agreement can.

The qualitative cases are the most convincing part for me. After training, the 4B student refuses to modify a basic-economy booking instead of rebooking it anyway. It also stops confusing the order it reads an address from with the order it should update, and it actually calls resume_line instead of promising to. These are the policy and follow-through failures I see most often in production agents.

Where I would be careful

The paper's own limitations are the right ones, and I would stress them:

• Agreement is not correctness. The student learns the teacher's mistakes as faithfully as its good calls. If your teacher has a known failure mode, ActionOPD will distill it. Pair the method with outcome checks on the tasks you care about.
• The matcher is a model. An LLM judge decides whether two messages agree, and its false positives and negatives become reward noise. I would audit a sample of match decisions before trusting a run, just as I would audit any LLM-as-judge signal.
• Three teacher samples is coarse. When many decisions are plausible, as in long-horizon or open-ended states, three samples may contain no match at all. A larger K costs more API calls, and the authors did not test whether it helps.
• The cost structure moves. You no longer need logprobs, but at every visited state you now pay for K teacher generations plus up to N × K student–teacher comparisons (12 at the paper's settings), any of which can fall through to an LLM matcher call. Caching teacher responses helps, and the paper does it. Budget for API usage at the state level, not per episode.

Why this matters for teams shipping agents

For years, distilling from a closed model meant imitating its transcripts. ActionOPD offers an on-policy alternative whose interface to the teacher is just generation. That changes the question you ask when choosing a teacher. You can pick on decision quality alone, without asking whether its tokenizer matches yours. Defining the action space is now the main engineering work: what counts as "the same call" and "the same message" in your domain. In regulated workflows that definition is often already written down, as tool schemas and policy rules.

The Self-ActionOPD extension, in which the same model acts as teacher with privileged task context, raised test pass^1 by 4.8 points. That is early evidence for action-level self-improvement, and it is the result I most want to see replicated at scale. The authors say code and checkpoints will be released at thu-nics/ActionOPD.