Scaling Laws Explained: What the OpenAI Paper Showed, and What Later Work Corrected

Kaplan et al. at OpenAI and Johns Hopkins reported in arXiv:2001.08361 (23 January 2020) that language-model loss falls as a power law in model size, dataset size and compute, with some trends spanning more than seven orders of magnitude. Their practical advice was to train very large models on modest data and stop well before convergence. Two years later, DeepMind's Chinchilla paper argued that this advice split compute the wrong way.

What did the scaling laws paper actually claim?

The paper, by Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei, reports that test loss follows smooth power laws. According to the abstract, width and depth have minimal effects within a wide range, so scale is what matters. These are the authors' own empirical findings from their runs.

Three fitted laws carry the paper, each valid when the other two factors are not the bottleneck (ar5iv full text):

In plain terms, doubling the parameters gives a loss smaller by a factor of 0.95, according to the paper. The gains are small per doubling but steady across a very wide range.

How was the evidence gathered?

Models ranged from 768 to 1.5 billion non-embedding parameters, trained on subsets of WebText2 from 22 million to 23 billion tokens, mostly with a 1024-token context (paper). WebText2 is an extended version of OpenAI's WebText, tokenized with byte-pair encoding and a vocabulary of 50,257.

That is small by later standards. The largest model in the study is roughly a hundred times smaller than the 175 billion parameter model described in the GPT-3 paper, which was submitted four months later and lists Kaplan as a co-author. The scaling-law conclusions about very large models are extrapolations.

What was the advice on spending compute?

The key result is how to split a fixed budget. The authors' empirical fits give optimal model size growing as C_min^0.73, batch size as C_min^0.24 and serial steps as C_min^0.03, so data grows only as about C^0.27. They write that extra compute should be spent "primarily on larger models, without dramatic increases in training time or dataset size."

A related rule of thumb from the same paper: every time model size grows 8x, data need only grow roughly 5x to avoid an overfitting penalty. This is where the "big model, modest data" reading comes from.

Did the paper make labs spend billions?

Direct causation has not been shown, so treat the headline version of this story with caution. What the paper did was give labs a forecastable relationship: more compute and parameters yield lower loss, with no deviation observed at the upper end of the tested range. The authors also say they expect larger models to perform better and be more sample efficient.

On cost, Epoch AI estimates that frontier training costs have grown 2 to 3 times per year for eight years, and that the largest runs could cost over a billion dollars by 2027 (an estimate published in 2024). Cost growth and the scaling-law result coincide, but that does not prove one drove the other.

The same group extended the idea beyond text. Henighan et al. (28 October 2020, 19 authors including Kaplan, Amodei, Radford and Schulman) report power-law-plus-constant scaling of loss in image, video, multimodal and math domains, with optimal-model-size exponents they describe as nearly universal across domains.

Where did Chinchilla disagree?

Hoffmann et al. (arXiv:2203.15556, 29 March 2022) trained over 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens. They conclude that model size and training tokens should scale in equal proportion, and that current large models were significantly undertrained. These are the paper's claims.

Their Table 2 compares the exponents directly (N_opt ∝ C^a, D_opt ∝ C^b):

To test it, the team trained Chinchilla at 70 billion parameters and 1.4 trillion tokens, against Gopher at 280 billion parameters and 300 billion tokens, with roughly the same compute. The authors report 67.5% average MMLU accuracy for Chinchilla, more than 7% above Gopher (lab-reported, as of the paper's March 2022 date).

Why do the two papers differ?

The authors of Chinchilla point to Kaplan's use of a fixed number of training tokens and one learning-rate schedule for all models, plus differences in scale: most of their models exceeded 500 million parameters, while most of Kaplan's runs were under 100 million.

A 2024 paper by Tim Pearce (Microsoft Research) and Jinyeop Song (MIT), arXiv:2406.12907 (published in TMLR 2024), offers a different primary cause. They find "much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale." At small sizes the embedding layers are not negligible, so a local power-law fit gives a steeper exponent. Simulating the Chinchilla setup under those conditions, they report local coefficients of 0.78 and 0.74, close to Kaplan's 0.73.

The two explanations are not mutually exclusive, and Pearce and Song say other differences, such as datasets and optimization, likely matter too. Their recommendation is that future studies count total parameters and compute.

Does the evidence hold up?

Several limits apply, and the authors name most of them.

• Tokenizer and dataset dependence. Kaplan et al. state that the fitted constants depend on vocabulary and tokenization and "do not have a fundamental meaning."
• Extrapolation. The authors note performance must eventually flatten out. Chinchilla's authors warn of significant uncertainty extrapolating many orders of magnitude, and that the frontier may curve at high compute.
• Thin large-scale validation. Chinchilla's direct test compares one model to Gopher at the largest scale. Runs were under one epoch of data, so repeated-data training is not covered.
• Reproducibility. Kaplan did not list model configurations, so Pearce and Song had to approximate them. Their own experiments are tiny (0.8M to 4.6M parameters on BookCorpus, context length 16), though their analysis code is public.
• Loss is not capability. These laws fit cross-entropy loss. They do not by themselves predict any specific downstream skill.

What does a working researcher take from this?

First, fit your own scaling curves before committing a large run, and report whether you count embeddings. Second, treat any "optimal ratio" as specific to a dataset, tokenizer and training recipe. Third, remember that data quality is a separate lever: Sorscher et al. (NeurIPS 2022) argue error can fall faster than a power law with data size if a good pruning metric exists, shown on ResNets for CIFAR-10, SVHN and ImageNet, not on language models.

What's still unclear?

Three questions remain open. Whether the Kaplan paper changed lab budgets, or simply described what labs were already doing, has no direct evidence behind it. How far the power laws hold at today's scales is an extrapolation, with the Chinchilla authors flagging possible curvature. And whether the Kaplan-versus-Chinchilla gap is mostly parameter counting, mostly training schedule, or both is argued by two groups with different primary explanations.