Students Beat Their Teachers: The Scaling Laws of On-Policy Distillation (2609.32722)

A 0.5B-parameter teacher that scored 39.8% on a math test taught a 14B student that peaked at 77.7% on the same test, according to arXiv 2609.32722. The paper, "Scaling Properties of Same-Family On-Policy Distillation" by Yuntai Bao and seven co-authors at Zhejiang University and Kuaishou Technology, was submitted on 26 September 2026. The title above is this site's framing: the headline claim holds for weak-to-strong pairs, and every number below is the authors' own, from one seed per run.

What problem does this paper solve?

It asks whether you can predict how well on-policy distillation (OPD) will work before you run it. The authors' central question is whether the performance of the student can be estimated from teacher scale and student scale alone.

OPD is a post-training method where the student generates its own answers and a teacher grades each token. Thinking Machines Lab contrasts it with the alternatives: supervised fine-tuning is off-policy with dense feedback, reinforcement learning is on-policy with sparse feedback, and OPD is on-policy with dense feedback. The lab also reports a 9 to 30x compute saving in one math example (lab-reported, and specific to its setup).

The practical hope is amortization. If a small RL-trained expert can teach every larger model in a family, you pay for RL once at small scale instead of at every size. The paper tests whether that holds and how much accuracy you lose.

How does it work?

The authors treat OPD like reward-model overoptimization studies: plot held-out accuracy against how far the student has drifted from its starting point. Their drift measure is d, the square root of the token-level reverse KL divergence from the student's initialization. They borrow the gold-score-versus-proxy-reward framing from Gao et al. (2023), but measure token-mean KL where Gao et al. measure sequence-level KL.

Models and data. The study uses Qwen2.5 Base models at 0.5B, 1.5B, 3B, 7B and 14B parameters. All get an initial SFT phase on a subset of Dolci-SFT. Teachers come from GRPO reinforcement learning on a mixed GSM8K and MATH training split of 14.8K examples, and everything is scored on a 6.3K-example test split. Each OPD run lasts at most ten epochs (580 updates), and the design has 25 teacher-student combinations, per the full text.

Two objectives. Vanilla-OPD rewards each token by the log-probability gap between teacher and student. Delta-OPD instead rewards the policy shift the teacher gained during RL: the teacher's log-probability minus that of the teacher's own pre-RL checkpoint. Delta-OPD is the shared core of several recent variants named in the paper (OPD², Direct-OPD, W2S-OPD).

The fitted laws. With N_S the student size in billions, N_T the teacher size (capped at the student's size) and G_T the teacher's score, the peak law is:

1 - G_peak = A * N_S^(-alpha) * N_T^(-beta) * (1 - G_T)^zeta

A matching law predicts the initial slope m of accuracy against d. The fitted Vanilla-OPD values are A = 0.97, alpha = 0.30, beta = -0.27 and zeta = 0.95. Delta-OPD gives 1.02, 0.34, -0.33 and 1.01. Because zeta is near 1, the student's remaining error is roughly proportional to the teacher's remaining error, scaled down by student size. The negative beta means that at the same teacher score, a smaller teacher transfers better.

What are the results?

Early in training, accuracy rises almost linearly in d. Across the 25 runs, linear fits to the first 30 checkpoints reached R² between 0.932 and 0.988. After that the curves turn noisy: some saturate, some keep improving slowly, some regress.

Source for all rows: the paper's text. Several findings stand out.

• Students pass weak teachers. The authors report that in every observed weak-to-strong pair the student's peak score exceeds the teacher's. The margin shrinks as teacher size approaches student size.
• Bigger teachers stop helping. Peak score improves with teacher scale only up to roughly the student's scale. The 3B student peaked with the 3B teacher, not the 7B one, and the 0.5B and 1.5B students lost accuracy under their largest teachers.
• Same-base OPD matches direct RL. Vanilla-OPD peaks with a same-size teacher sit within 0.5 percentage points of direct RL at all five scales, and exceed it at three. Direct RL on the student still beats every weak-teacher variant.
• Delta-OPD helps mostly with weak teachers. Its slope beat Vanilla-OPD in 15 of 17 shared cells and its peak in 12. With the 0.5B teacher it added two to four points; with larger teachers the gain was within about one point.

How well do the scaling laws predict outcomes?

They predict peak accuracy to within about one point at held-out scales, and teacher score is what makes them work. Table 2 of the paper reports leave-one-scale-out RMSE in accuracy points:

When the largest student or teacher is withheld, the joint law's error stays within 0.7 points for Vanilla-OPD and 0.4 for Delta-OPD.

The most informative test is a 7B student distilled from an intermediate 3B checkpoint (teacher score 66.0), slightly above the 1.5B endpoint's 63.6. The scale-only and score-only laws both favored the higher-scoring teacher. The joint law predicted 74.1 for the intermediate teacher; the observed peak was 73.8, below the 1.5B teacher's 77.5. That is the clearest evidence that a teacher's score alone does not define its teaching value.

The rate law is weaker. Its log-space R² values are 0.59 and 0.76, and the authors call it an interpretable summary, not a precise predictor.

Does bootstrapping through intermediate students help?

No. The authors chained OPD stages (0.5B expert, then 1.5B, 3B, 7B, 14B) and compared each stage's student with direct OPD from the 0.5B expert. Every chain peaked below the direct route.

The paper notes this reverses the bootstrapping gain that Burns et al. (2024) saw in a chess-puzzle setting. It also separates score from teaching value: the 1.5B OPD product scored 53.0% against the 0.5B expert's 39.8%, yet its 3B student peaked lower.

The same section tests supervision type. An off-policy SFT cold start cost 6.6, 15.7 and 19.4 points for the 3B, 7B and 14B students taught by the 0.5B expert. Pure off-policy distillation trailed OPD by 28.4 points in the most extreme weak-to-strong cell and by under one point in same-base cells.

Does the evidence hold up?

The scaling-law story is plausible, but the sample behind it is small. This section is the site's analysis, built on the paper's own limitations.

• One seed everywhere. The paper states that every SFT, RL and OPD run uses a single random seed, and that there are no prompt-bootstrap or optimization-seed intervals. A 0.5 point gap between two cells cannot be separated from seed noise with this design.
• Peaks are selected maxima. Gold score is accuracy on the test split, and the paper reports each run's peak over periodic evaluations of that same split. A maximum over many noisy checkpoints is biased upward, and the paper does not quantify that bias.
• Collinearity. On the RL-endpoint grid, teacher size and teacher score are almost perfectly correlated (r = -0.9999), so the exponents cannot be separated there. Only eight extra bootstrapped cells (five Vanilla, three Delta) lower it to -0.95 and -0.97 and identify the teacher terms.
• Narrow scope. One family (Qwen2.5), one math mixture, responses capped at 2K tokens. The authors say whether the coefficients transfer elsewhere is untested.
• Parameter count is confounded. The authors note it bundles model scale, data scale and compute, so the laws should be read at compute-optimal settings.
• Reproducibility. The paper uses the verl framework and lists hyperparameters in an appendix, but no code repository for this study is linked in the paper or on its arXiv page as of 16 October 2026.

None of this makes the results wrong. It means the fitted exponents are best read as a description of this grid, and the headline ordering findings (small teachers can win, bootstrapping does not help, cold starts hurt) are the parts most worth testing elsewhere.

How does it compare with related work?

It sits between three lines of research.

Pretraining distillation scaling. Apple's Distillation Scaling Laws paper (Busbridge et al., ICML) fits a law from compute and its split between student and teacher. It reports that a student can outperform its teacher, and that supervised learning is generally preferable when only one student is needed and the teacher must also be trained. Bao et al. study post-training on a fixed RL teacher, so the two ask different questions. Apple's page also notes that better teachers produce better students only up to a capacity gap, which is loosely in line with the finding here that bigger teachers stop helping once they outgrow the student.

Weak-to-strong generalization. Burns et al. (2024) showed weak supervision can elicit strong-model capabilities. Bao et al. extend this to on-policy supervision and find the bootstrapping benefit does not carry over.

Reward overoptimization. Gao et al. (2023) modeled gold reward against KL distance. Bao et al. borrow the framing but find OPD differs: after the linear phase the dynamics are heterogeneous, with no shared functional form. This connects to the question of what RL post-training does to a model, which our post on the Sharpening Tax covers from another angle.

For background on how scaling laws are fit and later corrected, see Scaling Laws Explained: Kaplan and Chinchilla vs Kaplan. For the RL side that produces the teachers, see DeepSeek-R1 Paper Explained.

What does a working researcher take from this?

Four practical takeaways, each limited to the paper's setting.

1. Pick teachers by teaching value, not only by score. At matched score, smaller teachers transferred better, and an OPD-produced teacher with a higher score taught worse than the weaker RL expert.
2. Do not cap yourself at the biggest teacher you have. Gains stopped near the student's own scale, and the smallest students got worse with much larger teachers.
3. Skip the off-policy cold start. In weak-to-strong setups it pinned the student near the teacher's score and OPD did not recover the loss.
4. Do not expect a chain to beat a direct step. Every bootstrapped chain peaked below direct OPD from the smallest expert.

Remember that direct RL on the student remained above every weak-teacher variant. The case for weak-to-strong OPD is cost amortization across a model family, not a higher ceiling.

What's still unclear?

• Does it generalize? No evidence yet beyond Qwen2.5 on GSM8K and MATH-style math. The authors list other families, tasks and post-training recipes as untested.
• How much is seed noise? With one run per cell and aggregate evaluation records, the paper cannot say. Multiple seeds and prompt-level resampling are the authors' own listed remedy.
• What is the mechanism? A local expansion explains why early transfer is linear in d, but the paper says the length of the linear window, the tails and the power-law forms remain empirical regularities without a mechanism.
• Safety. The authors' ethics statement warns that distillation can pass a teacher's hallucinations, biases and unsafe patterns to the student, and that bootstrapped weak-to-strong distillation may amplify them even while accuracy rises.
• Frontier scale. The largest student is 14B. Whether anything here holds at frontier size is not shown.

Read for this article: the full text of the arXiv HTML version of the paper, including its limitations and random-seed appendices. Results are the authors' claims and have not been independently replicated as of 16 October 2026.