CLIP Explained: The OpenAI Paper That Taught AI to See and Read

OpenAI's CLIP paper (arXiv 2103.00020, submitted 26 February 2021 by Alec Radford, Jong Wook Kim and ten co-authors) trained an image encoder and a text encoder on 400 million internet (image, text) pairs. The authors report that the best model reaches 76.2% zero-shot top-1 accuracy on ImageNet, matching the original ResNet-50 without using any of its 1.28 million labelled training images. The idea reshaped how vision models are trained, but the paper is also frank about where it fails.

All results below are the authors' claims unless noted as later, independent work.

What problem does the CLIP paper solve?

Standard vision models learn from a fixed set of labels, so they can only name categories someone annotated in advance. The paper's abstract says CLIP learns from raw text paired with images, and that natural language can then reference learned visual concepts or describe new ones, which lets the model transfer to tasks with no task-specific training.

Compared with the earlier zero-shot attempt the authors cite (Visual N-Grams), the paper reports ImageNet zero-shot accuracy rising from 11.5% to 76.2%. That jump is the headline. It is the same logic as GPT-3's few-shot prompting, applied to images: pretrain broadly, then specify the task in words.

How does CLIP work?

CLIP trains two encoders together: an image encoder and a text encoder. Both map their input into one shared embedding space. For each batch, the model is rewarded when an image's embedding is closest to the embedding of its own caption, and penalised when it is close to the other captions in the batch.

The reported setup, from Section 2 of the paper:

The text encoder is a transformer in the style introduced in "Attention Is All You Need", and the Vision Transformer applies the same architecture to image patches. The large batch matters because every other caption in the batch serves as a negative example.

To classify, you turn each class name into a sentence, encode it, and choose the closest text embedding. The paper found that wording matters: the prompt "A photo of a {label}." improves ImageNet accuracy by 1.3%, and ensembling 80 different prompts adds another 3.5%, for almost 5% together. The authors say this is similar to the gain from using four times more compute.

What are the results?

The authors benchmark on over 30 existing vision datasets. Key numbers as reported:

Margins vary widely. Zero-shot CLIP is ahead of the ResNet-50 probe by over 20% on Stanford Cars and Food101, but behind by over 10% on Flowers102 and FGVCAircraft. All figures are the authors' own evaluations.

The distribution-shift result is the most interesting. When the authors fit a logistic-regression classifier to ImageNet, ImageNet accuracy rose by 9.2% to 85.4%, but average accuracy under distribution shift slightly decreased (ImageNet-R fell 4.7%, ObjectNet 3.8%). Adapting to one dataset appeared to cost the model some of its general resilience to shift.

Does the evidence hold up?

Mostly yes on the core claim, with caveats the authors state themselves.

• Benchmark selection. The paper calls its 27-dataset suite "somewhat haphazardly assembled" and notes it was co-adapted with CLIP's development. Wins on 16 of 27 are not a random sample of vision tasks.
• Contamination. The authors checked 35 datasets for overlap with the training set. Nine had none detected, median overlap was 2.2%, average 3.2%. The largest was Country211 at 21.5%, with only a 0.2% accuracy gain. The maximum detected improvement was 0.6% (Birdsnap, 12.1% overlap). The analysis used RN50x64, so it does not directly cover the headline ViT.
• Reproducibility. Code and weights are public under an MIT licence, so third parties can test the claims, and they did (see below).
• Compute. The authors estimate around a 1000x increase in compute would be needed for zero-shot CLIP to reach overall state-of-the-art performance across their suite. Zero-shot is flexible, not cheap.

Where does CLIP fail?

CLIP fails on tasks that are fine-grained, abstract or far from its training data. The paper says it is poor at distinguishing car models, flower species and aircraft variants, struggles with counting objects, and can be near random on tasks like estimating the distance to the nearest car.

The sharpest example: CLIP reaches only 88% on handwritten MNIST digits, and the authors note that a logistic regression on raw pixels outperforms zero-shot CLIP there. Zero-shot performance was also weak on specialised sets such as EuroSAT, RESISC45, PatchCamelyon, CLEVRCounts, GTSRB and KITTI Distance.

Bias is documented too. On FairFace, 4.9% of 10,000 images were misclassified into non-human classes, with about 14% for images of Black people and under 8% for other groups. 16.5% of male images were misclassified into crime-related classes versus 9.8% of female images. Adding a "child" class sharply reduced these effects for people under 20.

How does CLIP compare with later work?

Two follow-ups test the paper's claims directly.

The Cherti paper adds a caveat to the original: the OpenAI and OpenCLIP models show different scaling behaviour despite identical architectures and similar training recipes, so the training distribution plays a key role. The data, not just the recipe, shapes the curve. For the general logic of such curves, see our explainers on Kaplan's scaling laws and the Chinchilla correction.

SigLIP changes the loss. The authors report that benefits of growing batch size quickly diminish and a batch of 32k is sufficient, against the 32,768 CLIP used at the time.

What does a working researcher take from this?

1. Prompts are part of the model. The 1.3% and 3.5% gains from wording and ensembling were measured on ImageNet. If you evaluate zero-shot, report your prompts.
2. Check zero-shot before fine-tuning. The paper's own result shows ImageNet adaptation can lower average accuracy under shift.
3. Do not assume it counts, measures or reads digits. Test on your task. The authors report poor results on counting and distance estimation.
4. Audit before deploying on people. The FairFace results show uneven error patterns, and class design (such as adding a "child" class) changed them.
5. Use open reproductions for research. The CLIP README itself points to OpenCLIP for larger independently trained models up to ViT-G/14.

What's still unclear?

• How much does training data drive the result? The Cherti paper shows data distribution changes scaling behaviour, but the original dataset's exact composition is described only at the level of queries and balance.
• Does contamination matter more for the headline model? The overlap study used RN50x64, not the ViT-L/14@336px model behind most reported results.
• How representative is the benchmark suite? The authors themselves flag the 27-dataset set as co-adapted with CLIP.
• Can bias be fixed by class design? One mitigation (the "child" class) helped for under-20s in the authors' test, but the paper treats it as a probe, not a solution.