GPT-3 Paper Explained: How "Language Models Are Few-Shot Learners" Changed Everything

"Language Models are Few-Shot Learners" (arXiv 2005.14165), submitted on 28 May 2020 by Tom B. Brown and 30 co-authors, mostly at OpenAI, reports a 175-billion-parameter language model that performs tasks from instructions and examples in its prompt, with no fine-tuning. In the authors' tests it reached 71.2% few-shot accuracy on closed-book TriviaQA and 86.4% on LAMBADA, while trailing fine-tuned systems by a wide margin on SuperGLUE. Every result below is the authors' claim unless a later paper is cited.

We read: the full text of v4 (arXiv HTML rendering) and the abstract page.

What problem did the GPT-3 paper try to solve?

It targeted the cost of task-specific fine-tuning. The authors argue that the dominant recipe needed a labelled dataset of thousands of examples for each new task, while people can pick up a task from a short instruction or a handful of examples.

Their bet was that scale could close the gap. The abstract describes the model as 10 times larger than any previous non-sparse language model, and tests whether a model that large can adapt through its prompt alone.

How does GPT-3 work?

GPT-3 reuses the GPT-2 architecture, a decoder-only transformer, with "alternating dense and locally banded sparse attention patterns" (arXiv v4, Section 2). The novelty is size, data and the evaluation protocol, not a new network design.

The paper trains eight models. All saw 300 billion tokens, and all use a 2,048-token context window.

Source: Table 2.1 of the paper. The paper lists eight sizes; five are shown.

The training mix, from Table 2.2, was weighted rather than proportional to dataset size:

Common Crawl began as 45TB of compressed plaintext and was cut to 570GB after filtering. The paper puts the data at 93% English by word count. Appendix D estimates training compute for the 175B model at 3.64E+03 petaflop/s-days, or 3.14E+23 flops.

What do zero-shot, one-shot and few-shot mean here?

The paper defines three prompt settings, all without weight updates:

• Zero-shot: a natural-language instruction only.
• One-shot: the instruction plus one demonstration.
• Few-shot: the instruction plus K demonstrations, where "we typically set K in the range of 10 to 100 as this is how many examples can fit in the model's context window (n_ctx=2048)."

This is the paper's lasting contribution. It turned prompting into an evaluation method, and it made "in-context learning" a research topic.

What were the results?

Results were uneven, and the paper says so. Here are headline numbers for the 175B model, all lab-reported:

Sources: Tables 3.2, 3.3, 3.7 and 3.8. The TriviaQA few-shot figure is from the wiki-split test server.

Translation showed a pattern. With 64 examples, few-shot BLEU into English was strong: 39.2 for French to English and 40.6 for German to English, against supervised results of 35.0 and 40.2. Translation out of English was weaker: 32.6 for English to French against a supervised 45.6, and 21.0 for English to Romanian against 38.5 (Table 3.4).

Arithmetic scaled with difficulty. Few-shot accuracy was 100.0% on two-digit addition and 80.4% on three-digit addition (the paper's prose says 80.2%, a small internal inconsistency), then fell to 25.5% at four digits and 9.3% at five (Table 3.9).

The result that drew the most attention was human detection of generated news. Participants identified articles from the 175B model at about 52% accuracy (95% CI 49 to 54%), against about 86% for a deliberately poor control model (Table 3.11). The study used roughly 80 US participants per model and 25 articles of about 200 words, so it measures short-article detection, not general fluency.

Does the evidence hold up?

Partly. Four points matter for a working researcher.

1. Training-set overlap. Section 4 reports that a filtering bug caused "only partial removal of all detected overlaps" from the training data, and retraining was not feasible. The authors built clean versions of each benchmark by dropping examples with a 13-gram overlap, and found that "although potential contamination is often high (with a quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly."

Two results carry asterisks. PIQA had 29% of examples flagged and a 3-point drop on the clean subset. Winograd had 45% flagged and a 2.6% drop. Several language-modelling benchmarks, including the Wikipedia sets from GPT-2, were omitted because the authors found them contained in the training data.

2. Weak spots the authors name. The paper says few-shot GPT-3 does "little better than chance" on WiC and ANLI. On ANLI, few-shot test accuracy was 36.8, 34.0 and 40.2 across the three rounds (Table C.1), against a chance level of about 33%. On RTE, only the largest model performed convincingly better than random.

3. Few-shot is not fine-tuning. The SuperGLUE gap (71.8 against 89.0) shows the headline "competitive" claim applies to selected tasks. The paper also notes GPT-3 outperformed a fine-tuned BERT-Large on four of eight SuperGLUE tasks with 32 examples per task, which is a fair way to describe the trade-off.

4. The mechanism is unknown. Section 5 asks whether few-shot prompting teaches a task from scratch or helps the model recognise tasks it saw in pre-training. The paper does not settle that.

The NeurIPS 2020 Best Paper committee cited the paper's "very extensive and thoughtful exposition of the broader impact" alongside its science. Peer recognition is not replication, though.

Why did it matter at the time?

It arrived four months after Kaplan et al.'s Scaling Laws for Neural Language Models (arXiv 2001.08361, 23 January 2020), which reported that loss falls as a power law in model size, data and compute. Several authors overlap between the two papers. GPT-3 read as the large-scale experiment of that argument.

The NeurIPS committee called it "a very surprising result that is expected to have substantial impact in the field, and that is likely to withstand the test of time" (NeurIPS blog, 7 December 2020). The paper shared the Best Paper Award with two other papers that year.

What came after GPT-3?

Two follow-ups changed the recipe.

Chinchilla (arXiv 2203.15556, 29 March 2022). DeepMind trained over 400 models and concluded that "current large language models are significantly undertrained." Their rule: for every doubling of model size, double the training tokens. Their 70B-parameter Chinchilla used 4x more data than the 280B Gopher at the same compute, and the authors report it outperformed GPT-3 (175B) on a large range of downstream tasks. GPT-3's 300B tokens for 175B parameters looks light by that standard. The Chinchilla result is itself the authors' claim.

InstructGPT (arXiv 2203.02155, 4 March 2022). OpenAI fine-tuned GPT-3 on labeler demonstrations, then with reinforcement learning from human feedback. In human evaluations on their prompt distribution, the authors report that outputs from the 1.3B InstructGPT model were preferred to those from the 175B GPT-3, "despite having 100x fewer parameters." Raw few-shot prompting turned out to be a weaker route to a usable assistant than adding human preference data.

What holds up today?

Judging from the papers above, and labelled as this site's analysis:

• Holds up: in-context learning as a capability and as an evaluation protocol. The zero/one/few-shot vocabulary is still how labs report results.
• Holds up: scale as a driver of capability, with the revision that data volume matters as much as parameter count.
• Revised: the idea that a bigger base model alone gives a better product. InstructGPT shows alignment data changed user preference more than 100x more parameters did.
• Weaker than it looked: the contamination analysis. The authors disclosed the bug and flagged affected benchmarks, but a 13-gram check is a narrow test of overlap.

What's still unclear?

• How much of few-shot performance is recognition versus learning. The authors raise this themselves in Section 5 and do not resolve it.
• True contamination effects. With partial filtering and a 13-gram check, the reported clean-versus-full gaps are an estimate of the effect, not a measurement.
• Human evaluation scope. The ~52% detection result covers about 200-word news articles and about 80 participants per model.
• Internal inconsistencies. The prose and tables differ on a few numbers (3-digit addition at 80.2% versus 80.4%).