LeCun vs Video Generation: Why H-JEPA Says Robots Don't Need to Imagine Pixels (2610.06805)

H-JEPA, submitted to arXiv on 5 October 2026 by Zhang, Terver, Rabbat, LeCun and Balestriero (arXiv:2610.06805), argues that a robot planner should predict in a compact learned latent space instead of generating future video. The headline result: on Visual AntMaze, a three-level hierarchy raises success from 18% to 73% while using less planner compute. Those are the authors' numbers, and the paper never tests a video generator directly.

What problem does this paper solve?

It targets long-horizon planning with latent world models, where a single predictor has to reason across very different timescales. The authors say existing task-agnostic JEPA world models predict and plan at a single timescale or with multiple horizons in one shared latent space. Over a long horizon, small prediction errors compound and the planning cost surface gets hard to optimise.

The fix is hierarchy. When factors in the data change at different speeds, the authors argue, higher levels can discard fast, unpredictable detail and retain slower task-relevant state. That is where the title's claim about pixels comes from: a leaf moving in the wind is irrelevant to getting an ant through a maze, so the planner should never have to predict it.

How does H-JEPA work?

H-JEPA trains a stack of action-conditioned JEPAs, each with its own observation encoder, action encoder and latent predictor, without reward or reconstruction objectives. Level 1 is LeWM, a ViT-Tiny encoder that summarises each image through its [CLS] token, plus a causal transformer predictor. LeWM itself is described in a March 2026 paper by Maes, Le Lidec, Scieur, LeCun and Balestriero as an end-to-end JEPA trained from pixels.

Higher levels use two-layer MLP state encoders and causal transformer predictors. Each level has a stride (2 in the simulated environments), so level 2 predicts in jumps of two level-1 steps, level 3 in jumps of four, and so on. Each level trains with a latent prediction loss plus SIGReg, a regulariser that prevents the latent space from collapsing. All levels train jointly, and gradients from upper levels flow into the lower encoders. The paper trains up to four levels.

Planning runs top-down by gradient descent:

1. The top level optimises its macro-actions to minimise the distance between its predicted final state and the goal.
2. Each lower level treats the states predicted above as subgoals and matches its own predictions to them.
3. Level 1 outputs the primitive actions the robot executes.

Decoders exist only for visualisation and are trained after the fact. They play no part in training or planning.

What are the results?

The clearest result is Visual AntMaze. Success is a percentage over 50 held-out tasks, mean plus or minus standard error over three seeds, according to the paper's Table 1 and Fig. 6 (the flat LeWM figure comes from the text and Fig. 6).

The last three rows matter. Stacking levels without the hierarchical planner does not help: the extra levels only pay off when the planner uses their subgoals. The authors also report that three-level H-JEPA reaches nearly twice the success of HWM, the closest prior hierarchical latent model, on AntMaze. On FourRoom and Push-T the two are similar, and on OGBench Cube the H-JEPA advantage is more moderate.

On real-robot data, the paper uses DROID with an added inverse-dynamics (IDM) loss. Hierarchy improves offline planning fidelity at lower planner compute, and removing the IDM loss drives performance to zero. Frozen-encoder baselines, the authors say, need orders of magnitude more compute for lower fidelity.

Does the evidence hold up?

Partly. Four things limit how far the result travels.

Sample size. Each environment uses 50 tasks and 3 seeds. One task is worth 2 percentage points per seed, so the AntMaze jump from 18% to 73% is far outside the reported error bars, but the four-level result (63.3 ± 7.7) shows how noisy the deeper configurations are.

Uneven gains. Push-T performs poorly at three or more levels, which the authors attribute to short episodes. The three-level and four-level models see only 58% and 14% of LeWM's training transitions per epoch there. On the manipulation datasets, frequency gaps below 2x mean there is little selective abstraction. The paper also notes that an upper-level cost need not improve planning.

Offline robot evaluation. The DROID evaluation uses 16 pick-and-place clips, scored offline and open-loop with a Frechet fidelity metric. The authors write that a next step is to test whether these gains translate to closed-loop control on a physical robot. They also say a fidelity gap shows one planner is better, not by how much. Only one camera is encoded per model, and the wrist camera is never used.

Reproducibility. This is a strength. The GitHub repository is MIT-licensed and its README links trained checkpoints and datasets, including 84 simulated-environment models (9.7 GB). That makes the results checkable in principle.

Does H-JEPA actually test the case against video generation?

No, and this is the gap between the headline and the paper. The related-work section says JEPAs predict future latent states, avoiding both reward and pixel reconstruction, and that hierarchical video models reconstruct observations and do not plan. Generative systems such as Genie and Sora are cited in the references but are not benchmarked.

So the comparison is hierarchy against no hierarchy, inside the latent-prediction family. What the paper does show is narrower: you can plan well, and cheaply, without a decoder in the loop. Whether a pixel-generating world model would do better, worse or the same on these tasks is untested here.

How does it compare with related work?

The sharpest contrast is with HWM: H-JEPA gives each level its own latent space, and the AntMaze gap suggests that matters there.

What does a working researcher take from this?

• If your planning horizon is long and the data has separated timescales, a hierarchy with per-level latent spaces is worth trying before scaling a single predictor.
• Keep the SIGReg term. In the DROID ablations, without it the effective rank collapses to 1 to 14 out of 384 dimensions and planning falls below the do-nothing floor.
• With real-robot video, budget for action labels. The IDM loss is needed, and the authors list learning from action-free video as future work.
• Do not assume more levels is better: AntMaze peaked at three, and Push-T collapsed beyond two.

What's still unclear?

The biggest open question is closed-loop control on a physical robot, which the paper leaves as future work. A second is the head-to-head against generative video world models, which nobody has run on these benchmarks in this paper. A third is whether the gains survive tasks without clean timescale separation, since the manipulation results are weaker.

There is also a small source discrepancy: the paper describes the DROID corpus as 74,530 episodes, while the repository README says 74,896. Treat the exact count as approximate.