Attention Is All You Need Explained: The 2017 Google Paper Behind the Transformer

"Attention Is All You Need" (arXiv:1706.03762) was first submitted on 12 June 2017 by eight authors at Google. It introduced the Transformer, a translation model built only on attention, and reported 28.4 BLEU on WMT 2014 English-to-German, more than 2 BLEU above the best earlier results including ensembles. This post decodes what the paper claims, how it works, and which parts of the evidence are strongest.

Reading note: this decode is based on the full paper text (arXiv HTML version), not the abstract alone.

What problem does this paper solve?

It targets the speed and long-range limits of recurrent sequence models. The paper says the dominant sequence transduction models were complex recurrent or convolutional networks with an encoder and a decoder, and that recurrence forces computation to run one position at a time.

The prior art matters here. In 2014, Sutskever, Vinyals and Le used a multilayer LSTM to encode a sentence into a fixed-size vector and another to decode it, reporting 34.8 BLEU on their test set. The same month, Bahdanau, Cho and Bengio conjectured that the fixed-length vector was a bottleneck and let the decoder soft-search the source sentence. Attention already existed. The Transformer's claim was that attention alone was enough.

The paper's complexity table (Table 1) frames the trade-off for a sequence of length n and representation size d:

Self-attention connects any two positions in one step and needs no sequential chain. The cost is the n² term, which grows quickly with sequence length.

How does the Transformer work?

Each layer lets every token compute a weighted mix of every other token's representation. The core operation is scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V, where queries, keys and values are learned projections of the input.

Several design choices are worth knowing:

• Multi-head attention. The base model runs h = 8 attention heads in parallel with d_k = d_v = 64, then concatenates them.
• Encoder-decoder stack. Both the encoder and decoder have N = 6 layers. The base model uses d_model = 512 and a feed-forward width d_ff = 2048.
• Positional encoding. Because attention ignores order, the model adds sinusoidal position signals: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and the matching cosine for odd dimensions. The authors hypothesised this would help the model attend by relative position.
• Regularisation. Residual dropout of 0.1 (base model) and label smoothing of 0.1.

The footnote on author contributions says Jakob Uszkoreit proposed replacing RNNs with self-attention, Noam Shazeer proposed scaled dot-product attention, multi-head attention and the parameter-free position representation, and Ashish Vaswani with Illia Polosukhin designed and implemented the first Transformer models. These are the authors' own accounts.

Training used WMT 2014 data: about 4.5 million sentence pairs for English-German with a shared vocabulary of about 37,000 tokens, and 36 million sentences for English-French with a 32,000 word-piece vocabulary. Batches held roughly 25,000 source and 25,000 target tokens. Training ran on one machine with eight NVIDIA P100 GPUs, using Adam with a 4,000-step warmup.

What are the results?

The big model reports 28.4 BLEU (English-German) and 41.8 BLEU (English-French) on newstest2014, as the authors' own numbers. The table below is from the paper's Table 2, with training cost in floating-point operations as estimated by the authors.

The base model, trained for 100,000 steps in about 12 hours on the eight GPUs, scored 27.3 on English-German, which is above every single model listed. The big model took about 3.5 days (300,000 steps). On English-French, the big model's 41.8 is above the listed single models but only just above the ensembles' 41.16 and 41.29.

For parsing, a 4-layer Transformer scored 91.3 F1 on WSJ section 23 when trained on the WSJ alone (about 40,000 sentences) and 92.7 F1 in a semi-supervised setup of about 17 million sentences. The paper says this beats all earlier reported results except the recurrent neural network grammar of Dyer et al., which reports 93.3 in its generative form.

Google's own blog post of 31 August 2017 says the Transformer needs less computation to train and is a better fit for modern hardware, "speeding up training by up to an order of magnitude". That is a company claim, and the post gives its BLEU figures only in charts.

Does the evidence hold up?

The core translation result is well documented, but the paper is a single-lab comparison on two benchmarks. Treat the numbers as the authors' claims.

What is strong:

• Ablations. Table 3 varies heads, layers, dimensions and dropout on the English-German development set (baseline 25.8 BLEU). A single head scored 24.9, and 4, 16 and 32 heads scored 25.5, 25.8 and 25.4. Two layers scored 23.7 against 25.3 for four. Removing dropout (0.0) dropped the score to 24.6.
• Positional encoding choice. Learned positional embeddings scored 25.7, which the paper calls nearly identical to the sinusoids. The sinusoidal design was a convenience, not a necessity.
• Code and data. The paper names a public codebase and standard WMT datasets, so the setup is inspectable.

What is thin:

• Cost comparisons are estimates. The paper multiplies training time, GPU count and an assumed sustained single-precision throughput per GPU. Baselines were trained on different hardware generations, so the FLOPs column is indicative, not exact.
• Single reported runs. The paper reports one BLEU figure per setting. Variance across seeds is not shown, so small gaps (for example 28.4 against 26.36) are less informative than the roughly 2 BLEU headline gap.
• Narrow scope. Only translation and one parsing task. The paper says the model generalises, but two tasks do not establish that.
• Dev-set tuning. Ablations use newstest2013 while headline results use newstest2014, which is the right separation, but checkpoint averaging and beam settings (beam 4, length penalty 0.6) were also tuned choices.

How does it compare with related work?

It replaced the sequential chain of recurrent models while keeping the encoder-decoder shape and the attention idea from earlier work.

These BLEU values come from different test sets and setups, so they should not be ranked against each other. The comparison in the Transformer paper itself is Table 2.

The most direct documented descendant is BERT, submitted on 11 October 2018 by Devlin, Chang, Lee and Toutanova. Its title names "Deep Bidirectional Transformers", and its authors report new state-of-the-art results on eleven NLP tasks, including a GLUE score of 80.5%. That is the BERT authors' claim, not a replication.

What does a working researcher take from this?

Three practical lessons come out of the paper's own tables.

1. Check the head count. Table 3 shows quality dropping with one head and also with too many (32), so head count is a tuned hyperparameter.
2. Watch sequence length. The O(n²·d) cost is the price of one-step connectivity. The paper's restricted self-attention row (O(r·n·d)) points at neighbourhood attention as an option for very long inputs.
3. Reproduce with care. The original library is tensorflow/tensor2tensor. It was archived by its owner on 7 July 2023 and is read-only, and its README calls it deprecated in favour of Trax. Its README says the base model on eight GPUs for 300,000 steps "should reach a BLEU score of about 28", a repository claim, which differs from the paper's 27.3 for the base model. Check the README settings against the paper before comparing numbers.

Data point for planning: the paper's base model trained in about 12 hours on one eight-GPU machine. That is the authors' report, on 2017 hardware.

What's still unclear?

• 41.8 or 41.0? The abstract and Table 2 say 41.8 BLEU for English-French. The paper's Section 6.1 text says 41.0, and the Google Research publication page also states 41.0 in its abstract. The arXiv abstract and the table are the figures used in this post.
• Training cost per language. Table 2 lists one cost figure per Transformer row, so it is unclear if it applies to both languages.
• Which ingredient matters most. The ablations vary one factor at a time. They do not show how much of the gain comes from attention itself versus training recipe choices such as label smoothing, warmup and checkpoint averaging.
• Long-sequence behaviour. The quadratic cost is acknowledged in the paper's own complexity table, but the experiments use sentence-length inputs only.
• Later influence. The paper itself reports only translation and parsing results. Connecting it to later systems requires the follow-on papers, such as BERT.

What's next?

The paper's own tables suggest the open directions: restricted or local attention for long inputs, better compatibility functions than the dot product (the authors observe that reducing the key size hurts quality, which suggests a more sophisticated function may help), and applying the architecture beyond text. Those were the authors' suggestions in 2017, and the follow-up literature has taken up several of them.