DeepSeek-R1 Paper Explained: How Reinforcement Learning Alone Taught AI to Reason

In DeepSeek-R1 (arXiv 2501.12948, DeepSeek-AI, 22 January 2025), a base model trained with reinforcement learning and no supervised fine-tuning raised its AIME 2024 pass@1 from 15.6% to 71.0%. The shipped R1 model, which adds a few supervised stages, reports 79.8% on the same benchmark against 79.2% for OpenAI's o1-1217. All of these are the authors' own numbers; this post covers what they show and where the evidence is thin.

We read: the full text of the arXiv v1 HTML version. Peer-review details come from Nature's news coverage, not the Nature paper itself.

What problem does this paper solve?

It asks whether a language model can learn long, self-checking reasoning from reward signals alone, without a large set of human-written reasoning traces. Earlier approaches leaned on supervised chains of thought, process reward models or search. The authors report that none of the process reward model or Monte Carlo tree search attempts worked well enough to be worth the overhead.

The practical target was a model comparable to o1 on math, code and science benchmarks, released with weights. That last part matters for researchers: the paper plus the MIT-licensed release gave the field something to inspect and replicate.

How does DeepSeek-R1 work?

R1-Zero applies GRPO directly to DeepSeek-V3-Base, with rule-based rewards. The paper describes two reward types: an accuracy reward (a boxed math answer checked deterministically, or code run against test cases) and a format reward that requires reasoning inside <think> tags.

GRPO removes the critic. For each question it samples a group of G outputs and scores each one. The advantage of output i is:

A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)

The policy is updated with a PPO-style clipped objective plus a KL penalty to a reference model. The paper says it avoided neural reward models for R1-Zero because they "may suffer from reward hacking" and complicate the pipeline.

The training template is deliberately thin. It asks for reasoning, then an answer, and the authors say they avoided content-specific constraints such as mandating reflection.

R1-Zero had weaknesses: poor readability and language mixing. So R1 uses four stages:

1. Cold start: thousands of long chain-of-thought examples fine-tune DeepSeek-V3-Base.
2. Reasoning RL: the same large-scale RL, plus a language-consistency reward (the share of target-language words in the chain of thought).
3. Rejection sampling and SFT: about 600k reasoning samples and about 200k non-reasoning samples, roughly 800k in total, for two epochs of fine-tuning.
4. RL for all scenarios: rule-based rewards for reasoning data, reward models for general helpfulness and harmlessness.

What are the results?

R1-Zero's AIME 2024 pass@1 went from 15.6% to 71.0%, and 86.7% with majority voting over 64 samples (cons@64), according to the paper's Table 2. The table below shows R1 against OpenAI's o1-1217 and DeepSeek-V3, all as reported by the authors.

Source: Table 4 of the paper. Pass@1 uses temperature 0.6, top-p 0.95 and a 32,768-token cap, with "typically between 4 and 64" samples per question. The paper does not state a single eval size across tasks.

Two things stand out. R1 roughly doubles V3 on AIME 2024 (39.2 to 79.8). But it is level with or behind o1-1217 elsewhere, and clearly behind on factual QA (SimpleQA) and Aider-Polyglot.

Does the evidence hold up?

Partly. The training gains are large and the release allows checking, but several limits apply.

• Borrowed baselines. The o1-1217 numbers come from official reports because the authors found the API hard to access from mainland China. The comparison is not same-harness.
• No contamination analysis. The paper has no decontamination section. It does restrict LiveCodeBench to problems from August 2024 to January 2025 and uses 10 Codeforces Div.2 contests.
• Noisy small benchmarks. AIME-style results rest on a handful of problems, so differences of a point or two (79.8 vs 79.2) should not be read as a ranking.
• Data not released. According to Scientific American's reprint of Nature's news report, DeepSeek did not publish its training data. The authors told referees that R1 did not learn by copying reasoning examples from OpenAI models. Lewis Tunstall of Hugging Face, who reviewed the paper, is reported as saying he cannot be fully certain, but that replication attempts by other labs suggest the recipe does not need such data.
• Cost figure. The same report says the Nature supplementary material puts R1's training cost at the equivalent of US$294,000, on top of roughly $6 million for the base model. That is a company-reported figure from one outlet.

Was the "aha moment" really caused by RL?

The paper shows an intermediate R1-Zero checkpoint writing "Wait, wait. Wait. That's an aha moment I can flag here." and describes the model learning to rethink with an anthropomorphic tone. An independent paper disputes the causal story.

Liu et al. (arXiv 2503.20783, Sea AI Lab, NUS and SMU) hosted DeepSeek-V3-Base themselves and found it already produces self-reflection keywords like "Aha" and "wait". R1-Zero reflects more often, but they found the behaviour is not positively correlated with higher accuracy.

They also report an optimization bias in GRPO: dividing by response length and by reward standard deviation artificially increases response length, especially for incorrect outputs. Their fix, Dr. GRPO, removes those terms. With a 7B base model and the fix, they report 43.3% on AIME 2024 using 27 hours on 8 A100 GPUs.

My reading (site analysis): RL clearly shifted the behaviour toward longer, more verified reasoning, but "RL created reflection from nothing" is not supported. RL appears to amplify what pretraining already contained.

How does it compare with related work?

The two papers mostly ask different questions. R1 asks whether the recipe works at scale. Liu et al. ask which parts of the story are due to RL and which to pretraining. They conflict only on the interpretation of the "aha moment".

Inside the R1 paper there is a useful internal comparison on a 32B base:

The authors conclude distillation is economical and effective, while smaller models that rely on large-scale RL may not even match distillation. The six distilled models (1.5B, 7B, 8B, 14B, 32B, 70B, built on Qwen and Llama) use SFT only. The paper reports no RL stage for them. Distill-Qwen-1.5B reaches 28.9% on AIME and 83.9% on MATH-500.

What does a working researcher take from this?

• Verifiable rewards are the lever. Math answers and unit tests need no learned reward model, which removes one reward-hacking surface.
• Do the cheap baseline first. If you have a strong teacher, SFT on its traces beat RL from scratch at 32B in the authors' test.
• Watch length. Response length grows during RL, and Liu et al. show part of that is a loss-normalization artefact. Check whether your length growth tracks accuracy.
• Do not use few-shot prompts. The paper says few-shot prompting consistently degrades R1's performance; zero-shot is recommended.
• Mind the licence and cost. DeepSeek's release note says code and models are MIT-licensed and that API outputs can be used for fine-tuning and distillation.

What's still unclear?

• Data provenance. Training data is unpublished, so the origin of the 600k reasoning samples cannot be audited. DeepSeek's statement to referees about OpenAI-generated data is a denial, not evidence.
• Contamination. There is no decontamination analysis, and benchmark leakage cannot be ruled out from the paper.
• RL versus pretraining. How much of R1-Zero's gain is new capability and how much is surfacing behaviour already present in V3-Base is not settled.
• Narrow strengths. R1 trails V3 on function calling, multi-turn, role-play and JSON output, shows no large software-engineering gain over V3, and may reason in English for non-Chinese, non-English queries.
• Peer-review detail. Nature's news report says R1 is "thought to be" the first major LLM through peer review. Reviewer count and timeline were not given.

What's next?

The paper's own list: better general capability, fixing language mixing, prompt sensitivity, and software-engineering RL, where long evaluation times slowed training. It suggests rejection sampling on software-engineering data or asynchronous evaluation.

The open research questions are sharper. Which base-model properties make RL work? Can the length bias fixes in Dr. GRPO change the cost curve? Can reasoning be rewarded without a checkable answer? The R1 paper gives a working recipe; the follow-up work is testing which ingredients carry the result.