Does On-Policy Distillation Really Distill? Purdue Finds Teacher Noise and a Teacher-Free Fix (2608.31046)

Yi Ding and Ruqi Zhang of Purdue University argue that on-policy distillation (OPD) may not need its teacher. In arXiv 2608.31046 (submitted 31 August 2026, revised 24 September 2026) they report that teacher advantages disagree with answer correctness for 30.6% of answer tokens with a 4B teacher and 50.6% with a 235B-A22B teacher. A teacher-free method they call OPSA then lifts Qwen3-1.7B from 13.44 to 48.85 Avg@32 on AIME24, against 32.08 for OPD with a Qwen3-4B-Instruct teacher (paper PDF).

All numbers below are the authors' own, from one lab, with no independent replication reported as of 15 October 2026. The headline "the teacher is just noise" is stronger than what the paper shows, and the sections below explain where the line sits.

Source basis: the full paper PDF and the v2 abstract page (24 September 2026). The headline figures in the abstract match the PDF text.

What problem does this paper solve?

It asks where the gains of on-policy distillation actually come from. OPD gives a student model dense, token-level feedback from a teacher, as an alternative to the sparse outcome rewards used in reinforcement learning with verifiable rewards (RLVR). But the teacher scores student-written text that it would not itself generate, so the paper says the reliability of that supervision, and the source of the improvement, was unclear.

OPD builds on earlier work on training with the student's own outputs, such as Generalized Knowledge Distillation (arXiv 2306.13649, 2023), and on the Thinking Machines Lab post by Kevin Lu dated 27 October 2025. If you know RL post-training from DeepSeek-R1 or RLHF from InstructGPT, OPD swaps the reward signal for a per-token teacher signal.

How does it work?

The paper's loss is reverse KL on student-sampled tokens. Each token gets an advantage equal to the teacher's log-probability minus the student's: A_i = log(pi_t / pi_s). Tokens the teacher finds less likely than the student does get negative advantages, and tokens it prefers get positive ones.

The authors then run four steps of analysis, all with a Qwen3-1.7B student:

1. Measure noise. They sample one correct and one incorrect response for each of 500 DAPO-17k questions. They count a teacher signal as noisy if the sign of its advantage on the final \boxed{} answer tokens disagrees with the verifier.
2. Filter and retrain. They compare OPD on all trajectories, only noisy ones and only clean ones.
3. Find which tokens matter. They train only on subsets of tokens ranked by student log-probability.
4. Replace the teacher. They swap teacher advantages for a single fixed value.

From that they build OPSA. It updates only the 20% of sampled tokens with the lowest student log-probability. Each gets a negative advantage between -0.5 and -1.0, scaled by the token's entropy relative to the other low-probability tokens in the same response (the paper's Eq. 5). The paper says OPSA needs no teacher, ground-truth rewards or reference answers.

What does the teacher noise look like?

The authors report that noise is substantial and grows with teacher size. Teachers were Qwen3-4B, 30B-A3B and 235B-A22B Instruct models (paper).

For the 235B teacher, 97.8% of correct-answer tokens and 96.6% of incorrect-answer tokens receive negative advantages. In other words, the largest teacher is almost always negative and barely tracks correctness. The authors attribute this to growing mismatch between student and teacher distributions.

The surprising step comes next. In the filtering experiment, OPD on all trajectories, on only noisy trajectories and on only clean trajectories converge to comparable AIME24 performance after a similar number of steps. The authors read this as the student not matching the teacher's behaviour.

What actually drives the gains?

According to the paper, two things: low-probability tokens and negative advantages. Among tokens in the student's rollouts, 29.2% have exactly zero advantage and 51.7% have a magnitude below 1e-4, because the student and teacher agree on tokens the student is already confident about. Training only on the highest-probability tokens gives no noticeable AIME24 gain, even when advantages are replaced by random values.

For the signal, they restrict training to the lowest-20% tokens and assign a fixed advantage with no teacher:

This echoes earlier work. Zhu et al. (arXiv 2506.01347) found that training only on negative samples improves Pass@k across the spectrum up to k=256 and often matches PPO and GRPO. The Purdue paper cites that result as motivation.

What are the results?

On Qwen3-1.7B, OPSA beats GRPO, TTRL, OPD and OPSD on three math benchmarks. These figures are Avg@32 from the paper's Table 3 (non-thinking mode, lab-reported):

The OPSA margin over OPD on AIME24 is 48.85 minus 32.08, or 16.77 points, which matches the abstract. Pass@32 at least doubles on all three benchmarks for this model (AIME24 exactly doubles, from 40.00 to 80.00).

OPSA also helps larger models, with smaller relative gains. Qwen3-4B moves from 23.33 to 62.08 on AIME24, and Qwen3.5-9B from 76.35 to 87.81. Out-of-domain gains are small: on Qwen3-1.7B, MBPP+ moves from 58.24 to 59.44 and GPQA-Diamond from 27.92 to 32.40.

On cost, the authors report a Qwen3-1.7B step time of 46.3 seconds for OPSA, 61.2 for OPD and 186.2 for GRPO, since OPSA needs neither a teacher forward pass nor group rollouts. Training used the slime framework on 8 NVIDIA H100 or H200 GPUs.

Does the evidence hold up?

Partly. The mechanism evidence is controlled and interesting, but several limits keep "OPD does not distill" from being settled.

• The 263% headline is measured from a weak starting point. The base model runs in non-thinking mode at 13.44 on AIME24. With thinking mode switched on, the same base scores 46.56, close to OPSA's 48.85. The paper itself says OPSA shifts the policy toward longer-form reasoning, and inference token counts rise from 4,457 to 23,205. So much of the gain looks like recovering reasoning behaviour the model already had.
• The authors do test the length confound. In Table 8 they append "wait" to GRPO and OPD traces to match OPSA's length, and those baselines stay at 32.81 and 31.67 Avg@32. That control helps, but it is a crude way to give baselines more thinking.
• The noise metric is narrow. It looks only at final-answer tokens, on 500 questions with one correct and one incorrect response each. A teacher that is nearly always negative scores about 50% noise by construction, so the 50.6% figure says "uninformative on correctness at answer tokens", not "wrong everywhere". A teacher can still be right about intermediate tokens, which the authors say are hard to assess.
• Small evaluation sets, no error bars. The diversity analysis uses 30 AIME24 problems, and Pass@32 values move in steps of 3.33 points, consistent with about 30 problems per benchmark. The tables report single values without confidence intervals or seed spreads. A one-problem swing is 3.33 points of Pass@32.
• One baseline teacher. The main OPD baseline uses Qwen3-4B-Instruct. The 30B and 235B teachers appear in the noise analysis, so a stronger-teacher OPD baseline is not in the headline table.
• The TTRL baseline is unusual. It was trained directly on AIME24 with its best checkpoint reported, yet it barely moves. That says more about TTRL than about OPSA.
• Narrow scope. Students are Qwen3 and Qwen3.5 models, trained on DAPO-17k questions, tested mostly on math. Code and open-domain QA gains are small.

Reproducibility is better than average: the authors link a public GitHub repository and a Hugging Face collection, as listed on alphaXiv.

How does it compare with related work?

Three nearby papers ask related but different questions.

The first two point the same way as Purdue's paper: much of the improvement in these setups does not come from the specific content of the supervision. The Columbia paper asks something else. It models noise formally and proves that on-policy interaction helps, which is compatible with Purdue's finding that students tolerate noisy teachers. It does not claim teachers are unnecessary.

What does a working researcher take from this?

Treat the paper as a strong ablation recipe, not a verdict on distillation. If you run OPD, three checks are cheap:

1. Add a teacher-free control. Train the same student with a fixed negative advantage on the lowest-20% log-probability tokens, and compare. If it matches your OPD run, the teacher may not be doing the work.
2. Measure sign agreement. Check how often teacher advantages on verifiable answer tokens agree with your verifier, as the paper does.
3. Report the thinking-mode baseline. If your student has a thinking mode, include it, since it shows how much of a gain is behavioural switching-on of existing skill.

Avoid fixed positive advantages. The paper reports collapse within about 40 steps with +0.2.

What's still unclear?

• Does the result hold beyond Qwen and math? The paper tests Qwen3 and Qwen3.5 students, mostly on competition math. Whether a student from a different family, or a task where the student truly lacks knowledge, still benefits from a teacher is open.
• Is the noise really harmful, or just uninformative? Students ignored it in these runs, but the analysis cannot separate "teacher is noise" from "student learns something else from the same signal".
• Is OPSA a better reasoning method, or a better way to elicit thinking behaviour? The thinking-mode comparison suggests the second, and the paper does not show OPSA stacking against a strong reasoning-trained teacher.
• Version differences. The figures here come from the PDF text. The v2 abstract of 24 September 2026 carries the same headline numbers; other differences between versions are not covered.
• Peer review. This is an arXiv preprint, and its arXiv page lists no peer-reviewed venue.

The fair summary: this paper shows that OPD's gains survive removing the teacher in one well-controlled setting. It does not show that distillation never transfers knowledge.