From AlexNet to ResNet: The Papers That Made Deep Learning Work

Between the 2012 and 2015 ImageNet challenges, top-5 error on 1000-class classification fell from 16.4% to 3.57%. The 2012 figure is the SuperVision entry (AlexNet) using only the supplied training data, and the 2015 figure is a Microsoft Research ResNet ensemble. Four years, a handful of papers, and one idea per paper: GPUs and non-saturating neurons, smaller filters, parallel branches, normalization, and shortcuts.

Every number below is the authors' own claim on one benchmark, and the conditions differ between entries (single model versus ensemble, one crop versus many). We flag those differences where they matter. If you want what came next, the story continues in Attention Is All You Need Explained and GPT-3 Paper Explained.

What was the benchmark that made these papers possible?

It was the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Classification covers 1000 object classes, about 1.2 million training images, 50 thousand validation images and 100 thousand test images, according to the challenge paper by Russakovsky et al. (arXiv 1409.0575). The underlying ImageNet database is far larger: the project site lists 14,197,122 images in 21,841 synsets (as of 13 October 2026).

The metric is top-5 error: a prediction counts as correct if the true label is among the model's five guesses, as the 2012 results page describes (the column is headed "Error (5 guesses)"). Everything in the table below uses it unless noted.

What did AlexNet actually show in 2012?

AlexNet showed that a large convolutional network trained on GPUs could beat hand-engineered pipelines by a wide margin. In the official 2012 results, SuperVision (Krizhevsky, Sutskever and Hinton, University of Toronto) scored 0.15315 with extra ImageNet data and 0.16422 with only the supplied data. The best entry from another team, ISI, scored 0.26172, a weighted sum of Fisher-vector classifiers.

The team's own description lists 60 million parameters, five convolutional layers, three fully connected layers, two NVIDIA GPUs and about a week of training. The NIPS 2012 paper (pp. 1097-1105) abstract credits non-saturating neurons, an efficient GPU implementation and a new regularization method (dropout) in the fully connected layers. On the 2010 test set the abstract reports 39.7% top-1 and 18.9% top-5 error.

"AlexNet" is the name the field adopted afterwards for the SuperVision entry. This section relies on the official results page and the paper's abstract, not the full paper text.

What did the 2013 analysis paper add?

It explained why the networks worked, and showed they could be improved by inspection. Zeiler and Fergus introduced a visualization technique for intermediate feature layers and ran an ablation study of the contribution of different layers (arXiv 1311.2901). The authors report that this guided them to an architecture that outperforms Krizhevsky et al. on ImageNet, and that the features transfer to Caltech-101 and Caltech-256 once the softmax classifier is retrained.

This paper matters less for a number and more for a habit: treat the network as something you can probe, not a black box that happened to work.

Why did VGG and GoogLeNet go deeper in different ways?

They answered the same question, "does depth help?", with opposite design philosophies. Both VGG (Simonyan and Zisserman, Oxford, arXiv 1409.1556) and GoogLeNet (Szegedy et al., Google and collaborators, arXiv 1409.4842) were ILSVRC 2014 entries, posted within two weeks of each other in September 2014.

VGG's argument is that two stacked 3x3 layers see a 5x5 region and three see 7x7, while adding extra non-linearities and using fewer weights than one 7x7 layer. GoogLeNet's argument is about compute: its authors set a budget of 1.5 billion multiply-adds at inference and used 1x1 convolutions to reduce dimensions before the expensive 3x3 and 5x5 layers.

Read GoogLeNet's headline carefully. A single model with one crop scored 10.07% on validation; 6.67% needed seven models and 144 crops per image. That is a legitimate competition setup, but it is not the cost of one forward pass.

Why did batch normalization matter so much?

Batch Normalization made very deep networks trainable with higher learning rates and less care over initialization. Ioffe and Szegedy report that it reached the same accuracy as the original model in 14 times fewer training steps (arXiv 1502.03167, February 2015), and that an ensemble reached 4.9% top-5 validation error (4.8% test). The abstract says it also acts as a regularizer, in some cases removing the need for dropout.

A companion paper from He, Zhang, Ren and Sun introduced PReLU activations and an initialization derived for rectifiers. It reported 4.94% top-5 test error, a 26% relative improvement over GoogLeNet's 6.66%, and called its result the first to surpass the human-level figure of 5.1% attributed to Russakovsky et al.

These two results are not directly comparable: 4.94% is a test error, while Batch Normalization's headline 4.9% is a validation error. Both abstracts compare against human raters, and both are claims on a single, fixed 1000-class task.

What problem did ResNet solve?

ResNet solved the degradation problem. He et al. write that with increasing depth "accuracy gets saturated" and then degrades, and that this "is not caused by overfitting": adding layers to a suitably deep model raises training error. In their ImageNet test, a 34-layer plain network reached 28.54% top-1 validation error against 25.03% for the same depth with shortcuts, while at 18 layers the two were nearly level (27.94% versus 27.88%).

The fix is a reformulation. Instead of asking stacked layers to fit a target mapping H(x), the block fits F(x) := H(x) - x, so the output is F(x) + x. Identity shortcuts add neither extra parameters nor computational complexity.

def residual_block(x):
    out = conv_bn_relu(x)   # first layer of the stack
    out = conv_bn(out)      # second layer, no ReLU yet
    return relu(out + x)    # identity shortcut, then non-linearity

For the 50, 101 and 152-layer models the authors use a three-layer bottleneck block (1x1, 3x3, 1x1 convolutions), which they say was chosen for practical considerations of training time.

What are ResNet's numbers against the earlier papers?

ResNet-152 reached 4.49% top-5 validation error as a single model, and the ensemble reached 3.57% on the test set. The table collects the figures the ResNet paper reports for single models (validation set unless noted).

For ensembles on the test server, the same paper lists VGG at 7.32%, GoogLeNet at 6.66%, PReLU-net at 4.94%, BN-Inception at 4.82% and ResNet at 3.57%. The abstract says ResNet won first place in ILSVRC 2015 classification, and also in ImageNet detection, ImageNet localization, COCO detection and COCO segmentation.

On cost, the 152-layer network is 11.3 billion FLOPs, against 19.6 billion for VGG-19. The authors describe it as 8x deeper than VGG nets but lower in complexity.

What does the whole run of results look like?

Top-5 error on ILSVRC classification fell by about 12.9 percentage points in three years, but the entries are not like for like. This table keeps each figure with its condition.

The measured gap in 2012 is itself telling: the first-place entry beat the second-best team by about 10.9 percentage points (15.3% versus 26.2%, comparing the extra-data run with the best other team). A result of that size is why the field changed direction.

What's still unclear?

Several things are open, and the ResNet paper admits one of them. On why plain networks degrade, the authors write "The reason for such optimization difficulties will be studied in the future." They also call it an open question whether stacked non-linear layers can asymptotically approximate the residual functions. So the fix is empirical, with the cause not settled in the paper.

• Comparability. Single-crop, multi-crop, single-model and ensemble results are mixed across these papers. The GoogLeNet single-model, single-crop figure (10.07%) is far from its headline figure (6.67%).
• Benchmark limits. Beating a 5.1% human figure on a fixed 1000-class task, where the metric allows five guesses, is a narrow result. This is our analysis, not a claim from the papers.
• Dataset sizes differ in the sources. The AlexNet abstract describes a 2010 training set of 1.3 million images, while the ILSVRC challenge paper says approximately 1.2 million.
• Reproducibility. The numbers here are lab-reported. We have not replicated any of them, and the sources above do not describe independent replications.
• Where ResNet is weakest. The authors report no advantage for single-layer residual blocks, and their reasoning for why shortcuts help is a hypothesis supported by experiments.

If you follow the thread into language models, the same pattern of scale plus a training trick continues in Scaling Laws Explained.