InstructGPT and RLHF Explained: The Paper Behind Human-Feedback Fine-Tuning of GPT-3

In "Training language models to follow instructions with human feedback" (arXiv 2203.02155, Ouyang and 19 co-authors, OpenAI, submitted 4 March 2022), human labelers preferred outputs from a 1.3B InstructGPT model to outputs from the 175B GPT-3, despite the 100x gap in parameters. That is the authors' result on their own prompt distribution, not an independent replication. The recipe behind it, supervised fine-tuning plus a reward model plus PPO, is what most people now mean by RLHF.

Basis: the abstract plus the introduction, method and results sections of the paper's HTML full text, not the PDF. Figures below are the authors' claims unless stated otherwise.

What problem does the InstructGPT paper solve?

It targets the gap between predicting text and doing what a user asks. The abstract puts it plainly: making language models bigger does not inherently make them better at following a user's intent, and large models can be untruthful, toxic or simply unhelpful.

GPT-3 was trained to continue internet text. A prompt like "Explain the moon landing to a six-year-old" could be continued with more questions rather than an answer. The paper's fix is not a bigger model but a different training signal: what people say they prefer.

How does RLHF work in InstructGPT?

The pipeline has three steps, as described in the full text: collect demonstration data and train a supervised policy, collect comparison data and train a reward model, then optimize a policy against the reward model using PPO. Steps two and three can be iterated.

Prompts come from labelers and from requests to the OpenAI API, and the data is over 96% English, according to the paper.

What is the reward model?

The reward model starts from the SFT model with its final layer removed. Labelers rank between 4 and 9 responses per prompt, and all pairwise comparisons from one prompt go into a single batch element to avoid overfitting. The authors used 6B reward models only, because 175B reward model training "could be unstable" and cost more compute.

What is the training objective?

The reinforcement learning objective combines three terms:

objective = E[ r(x, y) - beta * log( pi_RL(y|x) / pi_SFT(y|x) ) ] + gamma * E[ log pi_RL(x) ]

The first term is the reward model score. The second is a per-token KL penalty that keeps the policy near the SFT model. The third mixes in the original pretraining objective. The paper calls models trained with gamma set to 0 "PPO" and those with the pretraining term "PPO-ptx", and says that unless otherwise specified, InstructGPT refers to PPO-ptx.

What are the results?

On the authors' prompt distribution, labelers preferred 175B InstructGPT to 175B GPT-3 outputs 85 ± 3% of the time, and to few-shot 175B GPT-3 71 ± 4% of the time. The paper also reports 78 ± 4% preference over FLAN and 79 ± 4% over T0.

Compute is the quieter headline. The paper lists 4.9 petaflops/s-days for the 175B SFT model and 60 for 175B PPO-ptx, against 3,640 for GPT-3 pretraining. Alignment was cheap relative to pretraining.

What is the alignment tax?

Plain PPO regressed against GPT-3 on SQuAD, DROP, HellaSwag and WMT 2015 French-to-English. PPO-ptx mitigates the regressions on all datasets and even surpasses GPT-3 on HellaSwag, but it still lags on DROP, SQuADv2 and translation. Raising the KL coefficient did worse: validation reward dropped a lot and DROP and SQuAD never fully recovered.

Does the evidence hold up?

Partly. The preference result is strong in size but narrow in scope, and several caveats come from the authors themselves.

• The judges are the trainers. About 40 contractors hired through Upwork and ScaleAI wrote demonstrations and rankings. Training labelers agreed with each other 72.6 ± 1.5% of the time, and held-out labelers 77.3 ± 1.3%. Reward models predicted held-out labelers' preferences with 69.6 ± 0.9% accuracy, against 72.4 ± 0.4% for training-group labelers. That is a small drop, which suggests some generalization, not proof of broad agreement.
• The prompts are the lab's. Win rates are measured on OpenAI's own prompt distribution, largely from API users of earlier InstructGPT models. A different user base could rank differently.
• Safety gains are conditional. The toxicity advantage appears when the model is prompted to be respectful and disappears with no prompt. When explicitly prompted to be toxic, InstructGPT outputs are "much more toxic" than GPT-3's. Winogender and CrowS-Pairs showed no significant bias improvement.
• Simple mistakes remain. The paper lists accepting false premises, over-hedging, and degrading when instructions contain multiple constraints.
• Whose values? The authors write that the procedure "aligns the behavior of GPT-3 to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of 'human values'." Labelers were mostly English-speaking people in the United States or Southeast Asia.

Where did the idea come from?

InstructGPT combined two earlier lines of work. Deep reinforcement learning from human preferences (Christiano, Leike, Brown, Martic, Legg and Amodei, submitted 12 June 2017) defined goals through human preferences between pairs of trajectory segments, and reported feedback on less than one percent of the agent's interactions. Paul Christiano and Jan Leike also appear on the InstructGPT author list.

Learning to summarize from human feedback (Stiennon et al., submitted 2 September 2020, NeurIPS 2020) applied the pattern to language. The authors collected human comparisons between summaries, trained a reward model, and fine-tuned a summarization policy with reinforcement learning on Reddit TL;DR posts. They report that it significantly outperformed human reference summaries and much larger models trained with supervised learning alone. InstructGPT widened this from one task to a broad mix of instructions.

What came after InstructGPT?

InfoQ reported on 25 February 2022 that OpenAI had deployed InstructGPT models as the default on its API. InfoQ also relayed a warning: because the model follows instructions better, a malicious user could use it to be less truthful and helpful and more harmful.

On the research side, Direct Preference Optimization (Rafailov et al., submitted 29 May 2023) questions whether the full loop is needed. Its authors describe RLHF as complex and often unstable, and derive a closed-form policy so the problem can be solved with a simple classification loss on preference data. They report that DPO exceeds PPO-based RLHF at controlling sentiment and matches or improves quality in summarization and single-turn dialogue. As with InstructGPT, those are the authors' own experiments.

What does a working researcher take from this?

Four practical points follow from the paper.

1. Preference data can beat parameters. On the authors' evaluation, 100x fewer parameters plus human feedback won. Check your own task before assuming scale is the answer.
2. Budget for the regression. If you run RL against a reward model, measure public benchmarks before and after. The paper's PPO-ptx term exists because plain PPO lost ground.
3. Treat the labeler pool as a hyperparameter. Who writes and ranks the data sets what "good" means. The paper itself says alignment is to a specific group, so report who that group is.
4. Keep a held-out annotator set. The authors used held-out labelers, who did not take the screening test, to check generalization. Cheap to copy.

What's still unclear?

• Independent replication. The headline preference numbers come from the authors' labelers and prompts. They are not independently replicated in the sources cited here.
• How much each step contributes. The paper reports SFT, PPO and PPO-ptx variants, but readers cannot tell from the main text how much of the gain would hold with different labelers or a different reward model size.
• Reward model size. The authors used 6B reward models because 175B ones were unstable. Whether larger reward models would score better, if stabilized, is not answered.
• Whether PPO is necessary. DPO's authors report competitive results without it, which is a claim to test on your own data.
• Scope. The InstructGPT paper covers the models it names; this post makes no claim about how any other product was trained.