RT-2 Explained: The Google DeepMind Paper That Started the Robot Foundation Model Race

RT-2 is a Google DeepMind model that fine-tunes a web-trained vision-language model to drive a robot arm by writing actions as text. In the paper (arXiv:2307.15818, 28 July 2023), the authors report that it matches the earlier RT-1 on tasks seen in robot data and reaches 62% success on unseen scenarios against RT-1's 32%, per DeepMind's announcement. Those are the authors' own figures from about 6,000 evaluation trials.

What problem does RT-2 solve?

RT-2 tackles the generalization gap in robot learning: robot datasets are small and narrow, while web vision-language data is huge and varied. The authors ask whether a single end-to-end model can learn to map robot observations to actions while keeping the benefits of large-scale language and vision-language pretraining, as stated in the abstract.

The practical goal is a robot that handles objects and commands absent from its demonstrations. A policy trained only on robot data has no way to know what a "toy dinosaur" or a "bag about to fall off the table" is. A vision-language model does.

The paper names this class of models vision-language-action (VLA) models and presents RT-2 as one instance. The idea builds on the same transformer lineage covered in our Attention Is All You Need explainer and the scale-first thinking in Scaling Laws Explained.

How does RT-2 work?

RT-2 treats robot actions as just another language. The paper's Section 3.2 builds on the discretization used in RT-1. Each action is eight integers: a terminate flag, three end-effector position deltas, three rotation deltas and gripper extension. The seven continuous dimensions are split into 256 uniform bins.

The model emits those integers as text, separated by spaces. Training prompts take the form "Q: what action should the robot take to [task instruction]? A:". Two base models use different tricks to represent the 256 bins:

At inference, when the model is given a robot task, sampling is limited to valid action tokens. On ordinary vision-language prompts it can still output any text.

The key training detail is co-fine-tuning. Rather than fine-tuning on robot data alone, the authors mix robot trajectories with the original web data in each batch, raising the sampling weight of the robot data. The robot data is the RT-1 set, collected with 13 robots over 17 months in an office kitchen, per the paper.

What are the results?

On seen tasks RT-2 performs about like RT-1, and on unseen tasks it does roughly twice as well. The paper says both RT-2 variants give about a 2x improvement over RT-1 and MOO on the unseen average, and the DeepMind post gives the headline as 32% to 62%.

The emergent-skills tests cover symbol understanding (for example "move apple to 3"), reasoning (math, multilingual commands) and human recognition ("move the coke can to the person with glasses"). DeepMind's post lists examples such as "move banana to the sum of two plus one".

The authors also report a chain-of-thought variant. They fine-tuned it for a few hundred gradient steps with an added "Plan" step, and it chose a rock as an improvised hammer and an energy drink for a tired person. That evidence is qualitative: the paper describes it as initial evidence and reports no success rates.

Does the evidence hold up?

Mostly for generalization, less so for the reasoning claims. The strongest evidence is the ablation: co-fine-tuning beat fine-tuning on robot data alone at both model sizes, larger models generalized better, and a 5B model trained from scratch did very poorly, according to Section 4.3. That points to the web pretraining as the cause, not just the architecture.

The weak points are worth stating plainly:

• Narrow setting. All robot data comes from one office-kitchen setup built for RT-1, so "unseen" means unseen within that family of scenes and robots.
• Few baselines on the headline test. The emergent-skills evaluation compared RT-2 only with RT-1 and VC-1, not with every baseline used elsewhere.
• Chain of thought is anecdotal. Rollouts illustrate the behavior but no rate is reported.
• Authors' evaluation. The numbers come from the team that built the model, on its own task suites.

How does RT-2 compare with related work?

Later papers tested the same bet on more robots, in the open, and with different action heads. Open X-Embodiment pooled data and trained RT-2-X, OpenVLA released an open 7B model, and pi0 swapped token prediction for flow matching.

Open X-Embodiment gives the most interesting follow-up number for RT-2 itself. In its Table II, RT-2 (55B, trained on Google robot data) scores 27.3% on emergent skills, while RT-2-X (55B, trained on pooled robotics data) scores 75.8%. The authors call that about 3x. On the standard generalization test they are on par, 62% against 61%. The paper ran 3,600 evaluation trials across 6 robots overall.

OpenVLA's claim matters because RT-2-X was a closed baseline. The OpenVLA authors say existing VLAs are largely closed and inaccessible, and they release checkpoints, fine-tuning notebooks and a PyTorch codebase. Its 16.5% figure is again a self-reported comparison.

For the language-model side of this story, see GPT-3 Paper Explained: RT-2 applies a similar "one big pretrained model, adapt with data" logic to robots.

What does a working researcher take from this?

Three lessons transfer. First, the recipe is simple: if your base model has a tokenizer, discretize actions into tokens and train with a standard language-modeling loss. Second, keep the web data in the mix. The ablation says co-fine-tuning, not robot-only fine-tuning, drives the generalization gain. Third, treat scale as a real constraint.

On that last point, the authors report the 55B model runs at about 1 to 3 Hz via a multi-TPU cloud service, and the 5B model at around 5 Hz. They flag real-time inference as a potential bottleneck for high-frequency control and point to quantization and distillation. Binning also limits action resolution to 256 levels per dimension, which is one reason later work such as pi0 moved to flow matching for dexterous tasks.

If you build or evaluate robot policies, the data side matters as much as the model. Cross-robot datasets, task coverage and eval design decide whether numbers like these can be trusted.

What's still unclear?

Several things remain open. The authors state that the robot gains no new motions from web data, so semantic generalization does not imply new physical skills. A robot that correctly identifies a rock still needs a grasp it already knows.

The reasoning claims are thinly supported: chain of thought is shown qualitatively, and the "rudimentary reasoning" category is a small set of hand-designed instructions. How much of the emergent behavior reflects true reasoning versus pattern matching on web-seen concepts is not settled by this paper.

It is also unclear how much of RT-2's edge survives a different robot, scene and task distribution without cross-robot data. Open X-Embodiment suggests data diversity helps a great deal on emergent skills, since the same architecture rose from 27.3% to 75.8% once trained on pooled data. And reported comparisons between VLAs come from different authors, task suites and robots, so cross-paper rankings are weak evidence.

What's next for robot foundation models?

The direction since RT-2 is toward more robots, open weights and better action heads. The sources above show all three: pooled datasets with 22 robots, an open 7B model with released code, and a flow matching approach evaluated on long dexterous tasks. The unresolved questions are replication by independent labs, inference speed on onboard hardware, and benchmarks that make results comparable across papers.

Disclosure: Paper Huddle is affiliated with Humanbit, which works on data labelling, evals and voice and physical AI data. This post touches that market.