Subliminal Learning, Taken Further: One Teacher Word per Prompt Lifted HumanEval+ by 5 Points (arXiv 2609.29233)

A coding teacher that answers one ordinary word per prompt can pass part of its skill to a student, according to arXiv 2609.29233 by Ziyang Zhang, Yubin Jing, Yuanhao Zeng, Yuyao Li, Haofan Wang and Yichen Gong (Peking University, Georgia Institute of Technology, ShanghaiTech University, Tsinghua University and Lovart AI, submitted 24 September 2026). The authors report that 5,664 single-word teacher responses gave a 5.34 percentage point gain on HumanEval+ over an exact nuisance-matched control, using Qwen2.5-1.5B.

The paper is titled "Post-Training Leaves Behavioral Shadows on Unrelated Decisions". It extends subliminal learning from traits to capabilities. All numbers below are the authors' own, from a v1 preprint that has not been independently replicated.

What problem does this paper solve?

It asks whether a private post-training update leaves a trace in behavior that has nothing to do with the trained task, and whether a student can learn skill from that trace. Earlier subliminal learning work showed that traits move through unrelated data. This paper tests whether capability does too, using as little teacher output as possible.

The original result is the Nature paper by Cloud et al., published 15 April 2026. There, a teacher with a trait, such as favouring owls or broad misalignment, generated only number sequences, and a student trained on them picked up the trait. Anthropic's write-up describes the setup: the same initial model is fine-tuned on filtered teacher data. TechXplore's coverage says the effect mainly occurs when teacher and student are the same model.

The ATD authors call the unrelated-input differences a "behavioral shadow" of post-training. Their question has two parts: can the shadow be observed one decision at a time, and can learning from it transfer part of the teacher's target-task gain?

How does Active Taskless Distillation work?

ATD picks prompts where the public base model is almost indifferent between two ordinary words, asks the private teacher to choose one, and trains a student on the resulting prompt and word pairs. Near a tie, a small shift from the teacher's update can flip the choice, so each answer is a one-bit observation of that update.

The setup in the full text:

1. The public model M0 and the teacher share an ancestor. The teacher is M0 plus an unknown update delta.
2. Each candidate prompt offers two single-token words. The paper's example is a neutral story question followed by the options "jacket | tie".
3. Using only M0, the authors compute the pair-normalized probability q of the second word and keep prompts where |q - 0.5| is at most 0.02. The public model's top token must also be one of the two words.
4. The prompt set is fixed before any teacher response is seen. The teacher returns one greedy token per prompt. Responses outside the two candidates are discarded.
5. A student initialized from M0 is trained with ordinary cross-entropy on the retained pairs, with loss only on the response token.

The authors give a first-order argument for why this works: the choice is approximately the sign of the public logit gap plus the gradient of that gap dotted with the update. They test it empirically rather than prove it.

The primary coding setup used Qwen2.5-1.5B-Instruct as ancestor and student. The teacher was a frozen rank-16 code-DPO LoRA trained on 13,607 verified preference pairs from the KAKA22/CodeRM-UnitTest corpus (Apache-2.0), with benchmark overlap removed. The student was a rank-16 LoRA on a 157-step schedule. The acquisition used 17,858 near-tie queries and kept 5,664 rows (about 31.7%) over 128 distinct words. A frozen audit found no code, math, benchmark or task terms in those rows.

What are the results?

The student reached 51.22% on HumanEval+, matching the teacher's aggregate score, and beat every control arm. The headline 5 points is signal minus the exact-matched control, not signal minus the base model. Pass@1 is greedy, scored through EvalPlus on 164 tasks.

Source: Tables 1 and 2 of the paper. The base and teacher rows show signal minus that score. Authors' numbers.

Other results the authors report:

• Fresh teachers and carriers. Across five independently constructed acquisitions and three training seeds each, the aggregate effect was +4.80 pp, 95% CI [1.18, 8.74], with all 15 pairs positive.
• Adaptation regimes. The effect was +6.55 (rank-32 LoRA), +9.15 (rank-64 LoRA) and +4.67 (full fine-tuning), all with intervals excluding zero.
• Active versus passive. Spending the same teacher-query budget on ordinary prompts, without the near-tie filter, did worse: the active student was ahead by +4.27 pp at matched queries and +5.03 pp at matched rows.
• Per-problem agreement. On the 19 HumanEval+ tasks the teacher fixed, students solved 49 of 76 seed-task cells (64.47%) against 16 of 76 (21.05%) for the control.
• Other tasks. Six multiple-choice benchmarks, each with its own teacher, gained 0.81 to 2.90 pp over a shuffle control, all with intervals excluding zero.

Does the evidence hold up?

The coding result is better controlled than most distillation claims, but it is small and narrow. The paper's strongest evidence is one benchmark of 164 problems, one 1.5B model, four seeds and a lower confidence bound of 1.22 pp.

Sample size. One HumanEval+ task is 1/164, about 0.61 pp (my arithmetic). The 5.34 pp gain is roughly nine tasks on average, and the CI lower bound is about two tasks.

Controls are a strength. The exact control reuses the same prompts, the same multiset of teacher words and the same difficulty bins, and agrees with the teacher on only half the carriers. That rules out word frequency and generic fine-tuning as explanations. Note, though, that every control sits 2.6 to 3.4 pp above the base score of 43.29 (my arithmetic from the tables). Fine-tuning on near-tie carriers alone seems to help a little, so the comparison with the control is the right one, and the gain over the base (7.93 pp) mixes both effects.

Other models are weaker. The authors report positive mean HumanEval+ gains on Qwen3-1.7B, Qwen3-4B and Llama-3.2-1B, but the paired 95% confidence intervals include zero for all three.

Multiple-choice transfer is partial. The student recovers a small fraction of the teacher's gain. On ScienceQA the teacher gained 14.48 pp over base, while the student ended 1.73 pp above base (79.38 vs 77.65). On HellaSwag the teacher gap was 22.31 pp and the student moved 1.98 pp above base (66.82 vs 64.84). Those are my calculations from Table 2.

Reproducibility. The myboker/ATD repository releases the frozen training arms, carrier metadata and EvalPlus records for the four paired runs. A single command recomputes the paired bootstrap on a CPU and checks that the rounded result matches the paper.

When does transfer fail?

It failed in every case where the teacher's gain was not something the student could already express, or where the ancestors differed. The authors treat these as boundaries rather than footnotes.

The memorization case matters most for interpretation. Because the memorized-answer student did not improve, the authors conclude that ATD does not simply launder the teacher's answers. They read the pattern as: what moves is capability the student can be elicited to express, not the teacher's raw advantage. The MBPP+ result fits a low-bandwidth channel, where a small teacher gain leaves little to recover.

How does it compare with related work?

It sits between three lines of research, and the authors position it against each.

• Subliminal learning (traits). Cloud et al. in Nature used long teacher outputs, such as number sequences, to move traits. ATD uses one word per prompt and targets capability. Both need teacher and student to share an origin.
• Mechanism work. Blank, Bhatia, Rajamanoharan, Conmy and Nanda, in arXiv 2606.00995, report that subliminal learning is mediated by a single steering vector. ATD's functional diagnostics, which project the student's change onto the teacher's update direction, are a similar kind of analysis. The ATD paper reports that this projection tracks teacher update strength, with Spearman rho of 1 in 3 of 3 seeds on its code and SciQ trajectories.
• Distillation and model extraction. Standard distillation needs teacher probabilities or full responses. On-policy methods use student-generated trajectories. ATD assumes only a black-box one-token interface. The authors cite model extraction and fingerprinting work as related query-selection ideas.

For background on what ordinary post-training does to a model, see InstructGPT and RLHF Explained and Meta's "Sharpening Tax".

What does a working researcher take from this?

Treat it as a measurement result about how much a fine-tune shows up in innocuous outputs, and as a design lesson about query placement. It is not a practical way to distill a coding model.

• Query placement matters. Selecting prompts at the base model's decision boundary beat spending the same query budget on ordinary prompts, by +4.27 pp at matched queries.
• Use matched controls. The exact nuisance-matched permutation is worth copying when you test whether any distillation signal depends on prompt-label pairing.
• Check the ancestor assumption. The method relies on a shared public ancestor. If you ship fine-tunes of a public base, this paper is a reason to think about what black-box outputs reveal. That reading is my analysis; the authors frame it as observability of a private update.
• Wider answers help on some tasks. In an exploratory K=20 multi-token variant on Qwen2.5-0.5B, GSM8K gained +6.37 pp [4.37, 8.42], while HumanEval+ (+1.02) and ScienceQA (+0.84) had intervals reaching zero.

What's still unclear?

The main open question is scale. Every positive result is on models of 4B parameters or smaller, with LoRA or full fine-tuned students, and most on a single 1.5B model. Whether the effect survives at larger sizes, or with teachers whose update is not a DPO LoRA, is untested in the paper.

Other gaps:

• Statistical power. Four seeds and 164 tasks leave wide intervals. The three extra model families have intervals that include zero.
• Low bandwidth. The authors say the approach does not yield reliable transfer in every tested setting, and it needs actively constructed carriers and a known public ancestor. They describe ATD as an initial exploration.
• Why it fails on incompatible ancestors. The failing pairs differ in tokenizer, family and scale at once, so the authors call the ancestor requirement an empirical finding, not a proven condition.
• Independent replication. The released materials let a reader recompute the statistics from the authors' own run records. A retrain from scratch by another group has not been reported.
• Reading date. This post reflects the v1 preprint and repository as of 16 October 2026.

We read: the full paper text (HTML version of arXiv 2609.29233), including appendices, plus the GitHub README.