A 1.7B Model Gains 263% on AIME24 Without a Teacher: OPSA Explained (2608.31046)

Yi Ding and Ruqi Zhang of Purdue University report that a method called OPSA lifts Qwen3-1.7B from 13.44 to 48.85 Avg@32 on AIME24, a 263% relative gain, with no teacher, hints or verifiable reward. The paper is arXiv 2608.31046, submitted 31 August 2026 (v2 dated 24 September 2026). The result is lab-reported, and the percentage is large partly because the base model was evaluated with thinking mode switched off.

What was read: the full PDF (v1) and the v2 abstract page. The headline numbers match in both.

What problem does this paper solve?

It asks whether on-policy distillation (OPD) actually transfers knowledge from a teacher, or whether something simpler explains its gains. In OPD, a student model generates answers and a stronger teacher scores each token. Thinking Machines Lab describes it as scoring each step of the solution, punishing mistakes and reinforcing correct steps.

The practical pain is that OPD needs a teacher with a compatible vocabulary and white-box access to its logits. The authors write that this restricts teacher selection. Alternatives such as on-policy self-distillation (OPSD) swap the teacher for the model itself conditioned on hints like reference answers, which still needs extra annotation. The paper's answer is to remove supervision entirely.

If you want background on the reinforcement learning methods this builds on, see our explainer on the DeepSeek-R1 paper, which covers GRPO-style training with verifiable rewards.

How does OPSA work?

OPSA trains on only the 20% of sampled tokens with the lowest log-probability under the current policy. Each of those tokens receives a negative advantage whose size grows with the token's entropy. Nothing else is used: no teacher signals, ground-truth rewards or reference answers.

The paper builds up to this in three steps.

1. Measure teacher noise. The authors call teacher supervision "noisy" when the sign of the advantage on the final \boxed{} answer tokens disagrees with the verifier. They sampled one correct and one incorrect Qwen3-1.7B response for each of 500 DAPO-17k questions. The noise rate was 30.6% for a 4B teacher, 34.7% for 30B-A3B and 50.6% for 235B-A22B.
2. Test whether the noise matters. Training on only noisy trajectories, only clean ones, or all of them converged to comparable accuracy after similar numbers of steps.
3. Find what drives the gain. In OPD, 29.2% of tokens had exactly zero advantage and 51.7% had magnitude below 10^-4. Replacing all advantages with a fixed negative value on the lowest-20% tokens still improved the student. A fixed positive value collapsed training within the first 40 steps, with response length falling toward zero.

The final advantage, as written in the paper (Eq. 5), is:

A_i = -1/2 - (1/2) * (H_i - H_min) / (H_max - H_min)
applied only to tokens in the lowest 20% by log-probability

H_min and H_max are the minimum and maximum entropy over those selected positions within each response. So the advantage ranges from -0.5 to -1.0, which the repository README also states. In a pilot comparison, scaling the penalty up with entropy reached 50.0% Avg@4 on AIME24 versus 35.13% for standard OPD, and reversing the sign of the entropy relation was less stable.

The authors' interpretation is mechanical. A negative advantage on a sampled tail token pushes probability toward the head tokens. At high-entropy "fork" positions, penalising a sampled head token spreads mass among competing head tokens, which they say preserves diversity. Their related-work framing connects to Negative Sample Reinforcement, which found that training only on negative samples can match PPO or GRPO on Pass@k.

What are the results?

On Qwen3-1.7B, OPSA beats every baseline the authors ran, at every benchmark. All numbers below are lab-reported, from the paper's Tables 2 and 3, with thinking mode disabled unless stated.

The relative gains over the base model are 263.5% on AIME24, 264.4% on AIME25 and 307.2% on HMMT25 for Avg@32. Pass@32 more than doubles on all three. Averaged across the three math benchmarks, OPSA beats the best baseline by 11.04 points Avg@32 and 8.89 points Pass@32.

Other models, from Table 2 (AIME24 Avg@32):

Out-of-domain gains are small: on Qwen3-1.7B, MBPP+ moves from 58.24 to 59.44 and GPQA-Diamond from 27.92 to 32.40. Training used only the questions from DAPO-17k, with no labels.

Does the evidence hold up?

Partly. The mechanism analysis is clean and the ablations are informative, but the headline percentage needs context, and several evaluation details are thin. These are the points a careful reader should weigh.

The base is non-thinking. All training and the main evaluation disable thinking mode. The same Qwen3-1.7B with thinking switched on already scores 46.56 Avg@32 on AIME24. The OPSA model's 48.85 is 2.29 points above that, and the authors themselves say OPSA with thinking off is comparable to the base model with thinking on. A fairer reading is that OPSA teaches a non-thinking model to reason at length, rather than adding capability beyond what the thinking mode already had. On Pass@32 the non-thinking OPSA model is level with thinking mode on AIME24 (80.00) and lower on AIME25 (66.67 vs 73.33) and HMMT25 (50.00 vs 60.00).

Gains with thinking on are modest. With thinking enabled, OPSA adds 5.94 points on AIME24 Avg@32 (46.56 to 52.50). The authors' own limitations section says OPSA may not expand the exploration frontier, as reflected by the modest improvements in thinking-mode Pass@k.

Small benchmarks, single numbers. AIME24 Pass@32 values in the tables move in steps of 3.33 points (40.00, 70.00, 73.33, 80.00), which implies 30 problems, so one problem is 3.33 points. This is our inference from the table, not a stated figure. The tables report single values, with no error bars or seed counts reported.

Checkpoint selection. The appendix says the best checkpoint for each method is chosen by Avg@4 on a "validation set", and the training curves in the figures plot Avg@4 on AIME24. The text does not make clear whether that selection set is disjoint from the reported test benchmarks. Separately, TTRL was trained directly on AIME24, which is the authors' stated baseline design.

One OPD teacher. The main OPD baseline uses Qwen3-4B-Instruct as the teacher. The noise analysis shows larger teachers are noisier on this measure, but the benchmark comparison does not include OPD with the larger teachers.

Narrow noise analysis. The noise definition looks only at answer tokens inside \boxed{}, uses one student model, and samples 500 questions with one correct and one incorrect response each. It supports the "teacher signal is noisy" claim, but intermediate reasoning tokens are not measured.

Length is controlled for, once. The authors ran a token-budget control, appending "wait" to GRPO and OPD outputs to match OPSA's response length. Those baselines did not improve: 32.81 for GRPO and 31.67 for OPD against 48.85 for OPSA on AIME24 Avg@32. That is a useful check, though a one-token nudge is a weak proxy for a trained long-reasoning policy.

Reproducibility is good. Code is on GitHub under Apache-2.0, with checkpoints and training logs linked from the README. The setup is documented: slime 0.2.4, Megatron 0.16.0rc0, SGLang 0.5.14, 8 H100 GPUs, learning rate 1e-6, rollout batch 64, training temperature 1.0 with a 12,000-token cap, and evaluation at temperature 0.7, top-k 20, top-p 0.8, 32,768 tokens.

How does it compare with related work?

OPSA sits between three lines of work, and the paper's own Table 1 sorts them by which supervision they need.

• OPD. The Thinking Machines Lab post of 27 October 2025 is the OPD reference the paper cites. OPSA does not dispute that OPD works. It disputes the explanation for why.
• OPSD. Reinforcement Learning via Self-Distillation (arXiv 2601.20802) is one of the self-distillation papers OPSA cites. The OPSA authors report OPSD gains only appear when the student has thinking off and the teacher version has it on, creating a first-token mismatch.
• Other OPD critiques. Revisiting On-Policy Distillation (arXiv 2603.25562) studies OPD failure modes and fixes. These still rely on external supervision, per OPSA's related-work section.
• Negative-only RL. NSR, from Zhu et al. (arXiv 2506.01347), penalises incorrect trajectories using a verifier. OPSA's Table 7 reports NSR at 32.08 against OPSA's 48.85 on AIME24 Avg@32 for Qwen3-1.7B, with OPSA needing no verifier.

Efficiency is the other comparison. In the authors' Table 6 for Qwen3-1.7B, step time is 186.2 s for GRPO, 61.2 s for OPD and 46.3 s for OPSA. These are single-setup measurements by the authors.

What does a working researcher take from this?

Treat it as a result about OPD's mechanism first and a new recipe second. If you run OPD, the paper gives a cheap ablation: replace teacher advantages with a fixed negative value on the lowest-20% log-probability tokens and see how much of your gain survives. If most of it does, your teacher may not be contributing what you assume.

A few practical points:

• Check your baseline's decoding mode. A large relative gain from a low non-thinking starting point can shrink against the same model with thinking on.
• Log token-level advantage distributions. The finding that over half of OPD tokens had near-zero advantage is cheap to reproduce.
• Do not use a fixed positive advantage with this setup. The authors saw collapse within 40 steps.
• Keep the token fraction near 20%. Training on only the bottom 10% performed substantially worse in their ablation, while 20%, 30% and 40% all pushed Avg@4 above 45.
• Mask-test the mechanism. Excluding fork positions whose top candidates include reflective words such as "wait" or "but" largely removed OPSA's gains in length and accuracy.

OPSA's authors argue their entropy-scaled penalty neither narrows exploration nor degrades Pass@k, but the thinking-mode results are the weaker evidence. For why long reasoning traces matter at all, see chain-of-thought prompting.

What's still unclear?

• Scale. The authors tested up to 9B parameters and state it is unclear whether OPSA works on larger models or mixture-of-experts architectures.
• Already-sharp models. They expect benefits to be limited for heavily post-trained models whose distributions are already low-entropy. Qwen3.5-9B's smaller 15.0% AIME24 gain is consistent with that, though the paper does not claim it as proof.
• Why OPSA beats OPD. OPSA outperforms OPD with similar training dynamics. The authors say this calls for further theory to identify the true source of OPD's gains. The paper's evidence is empirical.
• Independent replication. The results are the authors' own. The released code makes replication feasible, but the paper reports no outside reproduction.
• Selection protocol. The paper does not identify which "validation set" picks the checkpoint.
• Title versus framing. The arXiv title asks whether OPD "really distills." The experiments support "teacher signal is noisy and a fixed negative signal reproduces much of the gain" on one student and one teacher family. They do not show that OPD never transfers knowledge in other settings.

What's next?

The authors suggest combining OPSA's token-level negative advantages with other reinforcement learning methods to widen exploration, and further theoretical work on OPD with the K1 estimator. The open empirical questions are the ones a lab with more compute could settle: a 30B or larger student, an MoE model, multiple seeds with confidence intervals, and a thinking-mode-first training run.

Until then, the safest summary is narrow. On small Qwen models, a teacher-free, entropy-weighted negative signal on the least likely tokens matches or beats teacher-based OPD and GRPO on three math benchmarks, at lower training cost, according to the authors.