Chinchilla vs Kaplan Scaling Laws: Were the First LLMs Trained Wrong?

Kaplan et al. (2020) said that as compute grows, most of it should go into bigger models: optimal parameters scale as C^0.73 and data as C^0.27. Hoffmann et al. (2022) said parameters and tokens should grow equally, C^0.50 each, and showed a 70B model on 1.4T tokens could beat a 280B model trained with the same compute. Later work suggests the gap came mostly from how Kaplan counted parameters, not from one team being careless. Every number below is the authors' own claim unless a later paper replicated it.

What did the Kaplan paper claim?

Kaplan et al. (arXiv:2001.08361, submitted 23 January 2020, OpenAI and Johns Hopkins) found that language model loss follows a power law in model size, dataset size and compute across many orders of magnitude. Their full text gives loss exponents of about 0.076 for non-embedding parameters (alpha_N) and 0.095 for dataset size (alpha_D).

The practical result concerned how to spend a fixed budget. The paper reports N ∝ C_min^0.73 and D ~ C^0.27, and the abstract concludes that compute-efficient training means "training very large models on a relatively modest amount of data and stopping significantly before convergence."

Two details matter later. First, N in the paper means parameters "excluding all vocabulary and positional embeddings." Second, the models were small: 768 to 1.5 billion non-embedding parameters, trained on datasets from 22 million to 23 billion tokens, mostly with a 1024-token context. The authors also note that their constants depend on the tokenizer and "do not have a fundamental meaning."

What did the Chinchilla paper claim?

Hoffmann et al. (arXiv:2203.15556, submitted 29 March 2022) trained over 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens. They estimated the optimal split three ways and report that all three disagree with Kaplan.

Source: Table 2 of the Hoffmann et al. paper.

Their test case was Chinchilla, 70B parameters on 1.4T tokens, against Gopher at 280B parameters on 300B tokens. The paper says both used the same FLOPs (5.76×10^23). The authors report 67.5% average accuracy on MMLU in the abstract (Table 6 lists 67.6% for 5-shot, against 60.0% for Gopher and 43.9% for GPT-3). These are lab-reported results, and Chinchilla also differed from Gopher in optimizer (AdamW vs Adam), tokenizer and data mix, as the paper itself notes.

How different are the two recommendations in practice?

Very different. Hoffmann et al. state that for a 10x larger compute budget, Kaplan's fit implies model size should grow 5.5x and tokens only 1.8x, while theirs implies equal growth of roughly 3.2x each.

Their Approach 1 table shows what the equal-growth rule looks like: 1B parameters with 20.2B tokens, 10B with 205.1B, and 67B with 1.5T. That is roughly 20 tokens per parameter, which became the popular rule of thumb. The Chinchilla paper's own framing is that models were too large for their data, not that scaling was wrong.

Why do the two papers disagree?

There are two main explanations, and they are not mutually exclusive.

Learning rate schedule (Hoffmann et al.'s explanation). The paper says Kaplan used "a fixed number of training tokens and learning rate schedule for all models." Hoffmann et al. found that setting the schedule to match the number of training tokens gives the best final loss regardless of model size. In their account, reading intermediate losses off a long cosine schedule overstates the loss of runs that would have finished earlier, which "eventually contributes to the conclusion that model size should increase faster than training data size."

Parameter counting at small scale (Pearce and Song's explanation). Pearce and Song (Microsoft Research and MIT, arXiv:2406.12907, v3 dated 21 November 2024) say "much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale." At small sizes, embeddings are a large share of total parameters, so the relation between non-embedding parameters and compute is not a clean power law. A local fit there gives a steeper exponent.

They tested this by simulating a Chinchilla-style study over 20 log-spaced models from 790 to 1.58B non-embedding parameters, then fitting on non-embedding parameters. They got exponents of 0.78 and 0.74 under two specifications, near Kaplan's 0.73. They also compared four optimization schemes in small runs and found the choice "has a smaller impact on scaling coefficients than switching" parameter definitions. That directly disputes the schedule explanation as the main cause.

Does the Chinchilla result itself hold up?

Mostly yes, with one real correction. Besiroglu, Erdil, Barnett and You at Epoch AI (arXiv:2404.10102, April 2024) tried to replicate the paper's third method by reconstructing 240 data points from a figure. Their results differ from the published values:

Epoch says the published Approach 3 estimates imply about 70 tokens per parameter, which conflicts with the roughly 20 used to train Chinchilla, while its refit implies around 20. It also says the reported confidence intervals were so narrow they would need over 600,000 experiments when fewer than 500 were likely run. According to Epoch, the Hoffmann authors confirmed that the cause was averaging rather than summing Huber loss values, which made the optimizer stop early.

Two limits on this critique. Epoch worked from points read off a plot, which limits loss precision to about 0.01 by its own estimate. And the critique targets only the third method. Methods 1 and 2 already gave about 0.5, and Pearce and Song's results are consistent with them.

Were the first LLMs trained wrong?

Not exactly. By the standard of the Chinchilla paper, a model like Gopher was larger than its compute budget warranted, but nobody in 2020 had the 2022 data, and Kaplan's own paper notes that in practice researchers "typically train smaller models for longer than would be maximally compute-efficient."

The better reading: Kaplan et al. measured something real at small scale, and the extrapolation to large budgets was biased by counting choices and training setup. The Pearce and Song result is useful for a researcher because it means the disagreement was largely a methodology artifact, which can be corrected, not a contradiction in how language models behave.

Is "Chinchilla-optimal" still the target?

No, because the objective changed. Chinchilla optimizes loss for a fixed training budget and ignores serving cost. Sardana, Portes, Doubov and Frankle (Databricks MosaicML, ICML 2024) add inference demand and conclude that "LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal."

In their worked example, a developer wanting a 13B-quality model with 2 trillion inference tokens can cut total compute by 17% by training a 7B model on more data. They trained 47 models from 150M to 6B parameters at 10 to 10,000 tokens per parameter and saw quality keep improving.

Meta's Llama 3 report shows both regimes. Its flagship has 405B parameters on 15.6T tokens (about 38.5 tokens per parameter, our arithmetic) with 3.8×10^25 FLOPs. Meta's fit suggested 402B parameters on 16.55T tokens, and it says the flagship is "approximately compute-optimal." Of its smaller models, Meta says it trained them "for much longer than is compute-optimal" and that they "perform better than compute-optimal models at the same inference budget." Those are Meta's claims.

What's still unclear?

• The cause of the gap is not settled. Hoffmann et al. point to the learning rate schedule; Pearce and Song dispute that and favor parameter counting. Pearce and Song call theirs the first-order cause and say other differences (datasets, transformer details, optimization scheme, how compute is counted) have a more minor effect.
• Pearce and Song's evidence is mostly simulation. Their own training runs use models of 0.8M to 4.6M parameters, and they assume the C = 6ND approximation and a fixed functional form between total and non-embedding parameters.
• Extreme overtraining is thinly tested. In Sardana et al., only the 150M model reached 10,000 tokens per parameter, larger ones reached up to 1,000, and the authors find models keep learning at extreme durations "more slowly than scaling laws predict." Fits on runs up to 100 tokens per parameter alone give very different exponents (alpha 0.08, beta 0.13) than fits on all data (0.18, 0.24).
• The Chinchilla paper has small internal inconsistencies. Its abstract reports 67.5% on MMLU while its Table 6 lists 67.6%, and the data range reads 5 to 500 billion tokens in the abstract but 5B to over 400B in the introduction.
• Single-lab fits. Almost every number here comes from one team fitting its own models. Independent replication exists for the Chinchilla fit (Epoch AI) but not at the scale of today's frontier runs.

What should a working researcher take from this?

1. Report which parameter count you use. Total parameters and total compute are what Pearce and Song recommend measuring, along with an offset term in the compute-loss relationship.
2. Match the learning rate schedule to each run's token budget before reading off a scaling frontier.
3. Treat 20 tokens per parameter as a training-compute optimum, not a deployment target. If you expect heavy inference, the Sardana et al. framework gives a different answer.
4. Check for small-scale bias. Pearce and Song see curvature in the non-embedding relationship below around 200M parameters, so a fit made only on models that small may not extrapolate.
5. Do not take published confidence intervals on scaling-law fits at face value without checking the optimizer and fitting details.