One query is almost enough: what on-policy distillation actually needs from data

I build post-training pipelines for a living, and one of the quiet assumptions baked into most of them is that more curated data is better data. On-policy distillation (OPD) — where a student model generates its own rollouts and a teacher supplies dense, token-level supervision on those rollouts — is exactly the kind of technique where teams spend real engineering effort curating diverse query sets, reasoning that broader coverage of the task domain should translate into a better student. A new paper, "Rethinking On-Policy Distillation of Large Language Models II: One Training Example" (Fu, He, Zuo, Huang, Zhang, Xiao, Qian, Luo, Gao, Wang, Liu, Ding, and Xiao), tests that assumption directly by starving OPD of data down to a single query — and the result should change how you think about what these pipelines actually need.

The experiment: one query, hundreds of steps

The setup is deliberately extreme. Instead of training OPD on a curated dataset, the authors pick a single training query and let the student generate rollouts against it, with the teacher scoring every token along the way, for hundreds of optimization steps. The finding: one-shot OPD keeps improving well past what you'd expect from a single example, and it recovers most of full-dataset OPD's performance gain — consistently, across task domains and across model families. This isn't a fluke on one benchmark or one model size; the effect holds up as a general property of the algorithm.

That's a strange result if you think of training data the way you would for supervised fine-tuning, where a single example gives you a single gradient signal and diversity is what prevents overfitting to that example. OPD behaves differently because the query isn't the only source of variation — the student's own sampling is.

State coverage: the metric that explains it

The paper's explanation rests on a metric it introduces called state coverage: the fraction of the states visited during full-data OPD training that a given (smaller) query set's rollouts also reach. A "state" here is a point in the generation process — a partial sequence the student has produced and needs to continue — so state coverage measures how much of the terrain the student would explore under the full dataset it actually explores under a restricted one.

The headline numbers are the core of the post: a single query already reaches 71.5% state coverage relative to full-data training, and most of that ground is covered within the first 100 steps. That's because a single query, sampled repeatedly under temperature, doesn't produce one rollout — it produces a distribution of rollouts, and that distribution alone touches a surprising majority of the states the student would ever need to visit. Adding semantically distinct queries pushes coverage further: at 16 queries, coverage reaches 98.9%, and validation accuracy matches full-dataset training. The two curves track each other closely, which is the paper's strongest piece of evidence that state coverage, not raw data volume, is the variable actually driving how good the student gets.

The image below lays out this contrast: coverage climbs steeply with just a handful of queries and saturates within the first 100 steps, while — as the next section explains — alignment to the teacher climbs at essentially the same slow rate whether you're training on one query or the whole dataset.

Data-overfed, algorithm-starved

Here's the twist that makes this more than a data-efficiency footnote. If state coverage saturates so fast, why does one-shot OPD keep improving for hundreds of steps instead of plateauing almost immediately? The paper's answer is that coverage and alignment are two separate bottlenecks, and only one of them is cheap.

Alignment — how closely the student's token-level behavior matches the teacher's supervision on the states it visits — converges slowly, and it converges at roughly the same rate whether training is run on one query or the full dataset. Even when you fix the set of states the student needs to absorb, it still takes hundreds of steps to actually absorb the supervision on them. So the bottleneck in OPD isn't exposure to enough of the task's state space — that happens almost immediately — it's the optimization process's rate of digesting the supervision it's already been given. The paper's framing for this is memorable: OPD is data-overfed but algorithm-starved. Rollouts expose broad supervision essentially for free; the slow part is the student's own convergence.

This reframes a lot of what curated OPD datasets are actually buying you. If your query set is large mainly to broaden state coverage, the marginal value drops off fast — 16 well-chosen, semantically diverse queries got this paper to within a rounding error of full-dataset performance, and the multi-teacher variant (MOPD) showed the same pattern: 16 diverse queries per domain matched full-data multi-teacher training. As a further stress test, the authors even tried content-light templates and off-domain WildChat queries as the training query set, and state coverage — and downstream performance — approached the real-query baseline anyway. Task content and the state coverage it induces can come apart; what matters for this mechanism is the states the rollouts touch, not necessarily how semantically relevant the query looks to the target domain.

What this means if you run an OPD pipeline

I'd draw three practical conclusions from this. First, if your OPD data pipeline is spending significant engineering effort on dataset curation and diversity, this paper suggests you should sanity-check whether coverage is actually the bottleneck for your setup — it's plausible that a much smaller, deliberately diverse query set gets you 95%+ of the benefit at a fraction of the data cost. Second, since alignment convergence is the slow, non-negotiable part regardless of data volume, the real lever for making OPD faster or cheaper is step efficiency of the optimization itself, not dataset size — which is exactly where the authors point future work. Third, the fact that off-domain and content-light queries can approach real-query performance is a caution against assuming your curated data is doing what you think it's doing; it's worth directly measuring state coverage against a small validation set rather than assuming that more (or more topically relevant) data automatically means better coverage.

None of this means data curation is worthless — the paper still shows a real, if shrinking, gap between one query and a diverse set of sixteen. But it relocates where the value of that curation comes from: not volume, but how efficiently a small set of queries can span the states the student needs to see. For teams running OPD in production post-training, that's a much cheaper thing to optimize for than we've generally been treating it as.