Fei-Fei Li's RoboRender: Sim-to-Real Jumps From 10% to 71% (on 10 Rollouts per Task)

RoboRender (arXiv 2610.09254, submitted 7 October 2026, cs.RO) is a paper by Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei and Jiajun Wu. The authors report that policies trained only on RoboRender-generated video reach a 71% average success rate on real robots, about 7.1x raw simulation renderings and about 3.6x conventional visual domain randomization. The catch is the sample: according to the authors' project page, that average covers eight tasks with 10 rollouts each.

Basis for this decode: the arXiv abstract and the authors' project page. The full PDF was not read, so details that appear only in the paper body are not covered here.

What problem does RoboRender solve?

It targets the visual gap between simulation and the real world. Simulation makes robot data cheap and gives exact action labels, but policies trained on simulated pixels often fail on real cameras. According to the abstract, common fixes use intermediate representations that can discard semantic information or need extra perception modules at deployment.

The classic fix is domain randomization: randomize textures and rendering so the real world looks like one more variation. Tobin et al. (arXiv 1703.06907, 2017) trained an object detector accurate to about 1.5 cm using only simulated images with non-realistic random textures. RoboRender takes the opposite bet: make the training pixels look real instead of deliberately unreal.

How does RoboRender work?

It turns each simulated trajectory into photorealistic RGB video while keeping the simulator's geometry, robot motion and action labels. The authors fine-tune a video model, built on Wan2.1-T2V-1.3B, that takes three inputs: simulated depth video, a language instruction and robot RGB mask video. The generated video is then paired with the simulator's states and actions to train a policy, which is deployed zero-shot on real hardware.

Details from the project page, all lab-reported:

• Video training data: about 126,950 real robot samples at 416x240, from AgiBot-World-Alpha (69,910 samples) and DROID (57,040), covering 39,692 scenes.
• Preprocessing: 81-frame windows across camera views, depth from Video-Depth-Anything (plus FoundationStereo where stereo exists) and robot masks from SAM3.
• Multi-view and long rollouts: views are stacked into one latent so objects stay consistent between external and wrist cameras. Long rollouts are generated autoregressively, chunk by chunk.
• Simulation data: trajectories come from a scripted task-and-motion pipeline built on MolmoSpaces and CuRobo, using precomputed antipodal grasp libraries. Only successful episodes are kept.
• Policy: a fine-tuned pi0.5 that predicts 32-step action chunks and executes the first 16, trained for 40k steps at batch size 72.
• Speed: TeaCache cuts a 3-view, 81-frame clip from 68.8 s to 35.3 s on one H200. DMD2 distillation reduces sampling from 50 steps to 6 and renders a clip in 4.29 s.

What are the results?

The headline is a per-task table where RoboRender wins every row. All numbers are the authors' own, in a natural office with varied robot starts and object locations, across three embodiments (Droid, R1Pro for mobile manipulation, and Yam). Each cell is a success rate over 10 rollouts.

The averages check out: the RoboRender column sums to 570 points over eight tasks, or 71.25%. The authors also report two comparisons beyond the headline:

• Other video models, same simulated trajectories (drawer / fridge): Wan-Fun-Control 0% / 0%, Cosmos-Transfer2.5 30% / 0%, RoboRender 90% / 70%.
• Depth-only policies that estimate depth from RGB at inference: 30% vs 70% on Yam, 25% vs 68% on Droid pick-and-place, and 10% vs 70% on R1Pro.

On video quality, using 500 held-out DROID and AgiBot clips, the project page reports FID 25.7 and FVD 93.9 for RoboRender. The same table lists Cosmos-Transfer2.5 at 42.8 and 164.5, RoboTransfer at 52.8 and 275.4, and Wan-Fun-Control at 62.5 and 267.8 (lower is better). The page does not say which generation setting that table covers.

The scaling result is stated in the abstract: performance improves with more generated videos per simulation trajectory, raising opening-task success by 65 percentage points. The baseline and final values appear only in a chart, so treat the size of the effect as unconfirmed until you see the figure.

Does the evidence hold up?

The direction is believable; the size is uncertain. Ten rollouts per task means each task score moves in 10-point steps, and a single success or failure changes it by that much. Pooled, each method has 80 trials. By our own arithmetic, a normal-approximation 95% interval around 57 of 80 successes is roughly plus or minus 10 points, so "71%" is better read as "somewhere near 60% to 80%".

Other points a reviewer would raise:

• A tiny denominator inflates the ratio. Raw simulation scores 0% on six of eight tasks, so "7.1x" mostly reflects a 10% base. The absolute gap, 61 points, is the more honest number.
• Domain randomization is one baseline. The paper's 3.6x is against one randomization setup. A stronger or better-tuned randomization recipe could narrow the gap, and the project page gives no sensitivity study.
• No real-data comparison on the headline. The page does not show how a policy trained on real demonstrations would score on the same eight tasks, so we cannot say how much of the real-data gap is closed.
• Single environment. All evaluation is in one office. The page gives no cross-site tests.
• Self-reported. The numbers come from the authors, and no independent replication exists yet.
• No limitations section on the project page, and no ablation of mask conditioning or the distillation step.

In its favour: the evaluation is on physical robots, includes three embodiments, and the authors state that R1Pro and Yam were unseen during video-model training. That is a meaningful cross-embodiment signal, though still at 10 rollouts per task.

How does it compare with related work?

RoboRender sits in a growing family of video-based augmentation methods that keep simulator geometry and regenerate the appearance.

These numbers are not comparable. They use different tasks, robots, metrics and baselines. The only like-for-like comparison is the one inside the RoboRender paper, where the authors run Cosmos-Transfer2.5 and Wan-Fun-Control on the same simulated trajectories and report lower success for both.

If you follow robot world models more broadly, our decodes of NVIDIA Long-WAM and Yann LeCun's H-JEPA ask a different question. They use video prediction to plan or remember; RoboRender uses video generation to manufacture training data.

What does a working researcher take from this?

Treat RoboRender as evidence that appearance, not geometry or dynamics, can be the bottleneck for sim-to-real, and that a generative model can fix it without a perception module at test time. Practical points:

1. Conditioning on depth plus a robot mask is the design choice to copy. It keeps the policy's action labels valid while freeing the model to invent textures and clutter.
2. Generate many videos per trajectory. The reported scaling is in videos per simulation trajectory, which is cheap relative to collecting more trajectories.3. Budget the rendering cost. The reported 4.29 s per 3-view clip applies to the distilled sampler on one H200. The undistilled path is about 35 s with caching.
3. Run more than 10 rollouts if you reproduce it. At 50 or more per task, the intervals tighten enough to see which tasks truly improved.
4. Check the code first. The project page links a GitHub repository and a Hugging Face models page. Confirm which checkpoints and licences are actually available.

What's still unclear?

• How well does the 71% hold up? With 10 rollouts per task and one environment, the interval is wide, and nothing shows how the gain holds under new lighting, new sites or new objects.
• What did the 65-point scaling gain start from? The abstract gives the size of the gain on opening tasks but the baseline and endpoint sit in a chart.
• How does it compare with strong randomization and real data? The headline compares against raw simulation and one randomization setup only.
• Which parts matter? The page reports no ablation of mask conditioning, multi-view stacking or distillation.
• What breaks it? The page has no limitations section, so failure modes such as contact-rich tasks or deformable objects are unreported.
• Is it replicated? Not yet, as of 13 October 2026.

What's next?

The obvious tests are independent reproduction on other robots, many more rollouts per task, and comparisons against tuned domain randomization and against real-demonstration policies. The cross-embodiment result on R1Pro and Yam, which the video model never saw in training, is the most interesting thread to pull. If it survives larger evaluations, video generation becomes a general-purpose bridge between cheap simulation and real cameras.