Yann LeCun's H-JEPA: 73% vs 18% Without Generating a Single Pixel (2610.06805)

A three-level hierarchy of latent world models reaches 73.3% success on Visual AntMaze, against 18.0% for the flat model it extends. That is the headline of H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning (arXiv 2610.06805, posted 5 October 2026) by Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun and Randall Balestriero, from AMI, NYU, INRIA Paris and Brown. The gain comes with less planner compute, according to the authors, and no pixel is generated at any point. The result is strong on mazes and weak on Push-T, and that split is the more useful part of the paper.

All results below are the authors' own, not independently replicated.

What problem does H-JEPA solve?

It targets long-horizon planning with a learned world model. A flat model that predicts the next few steps gives a poor signal for a goal 80 steps away, so the planner has little to follow. The abstract says existing task-agnostic JEPA world models plan at one timescale, or across horizons inside one shared latent space.

LeCun proposed a hierarchical JEPA in 2022, according to 36Kr. This paper is an implementation that trains from scratch.

How does H-JEPA work?

H-JEPA stacks action-conditioned JEPAs, each predicting in its own latent space at a longer stride, and trains all levels together (full text).

• Level 1 is LeWM, the flat baseline: a ViT-Tiny encoder and a causal-transformer predictor. One step spans 5 environment steps. LeWM is described in arXiv 2603.19312.
• Upper levels use a two-layer MLP encoder on the level below. A small transformer pools the two intervening action embeddings into a 4-dimensional macro-action. Every upper level has stride 2 (3 on DROID), so each step spans twice as many environment steps as the level below.
• Loss per level is one-step latent MSE plus a weighted SIGReg term, the Gaussian regularizer from LeJEPA that prevents representation collapse. There is no stop-gradient, no EMA target and no pretrained weights.
• Gradients from upper levels flow through the lower encoders, which is what "end-to-end" means here.

The planner is top-down and gradient-based. The top level minimizes the distance from its predicted final state to the goal latent. Its first predicted state becomes the subgoal for the level below, which matches its own rollout against that subgoal in the upper latent space, and so on down to primitive actions. Optimization uses AdamW for at most 90 iterations, and the agent replans every 10 environment steps.

The predictors are transformers, the same family covered in Attention Is All You Need Explained. What is new is the stacking and the way the goal cost is defined, not the block.

What are the results?

Success rates below are mean +/- standard error over three seeds, 50 start-goal tasks per environment. "Flat" is LeWM. Hierarchical means the full top-down planner. Source: Tables 1 and 9 of the paper.

Two points from the text. On AntMaze, three-level H-JEPA reaches nearly twice the success of HWM, an earlier hierarchical world model that plans in one shared latent space, according to both the paper and VentureBeat. The planner compute claim is harder to check. The only absolute figure in the text is the 100 TFLOP-per-episode budget used to pick the operating point. The compute-versus-success curves are in figures without tabulated values.

Does the evidence hold up?

Partly. The AntMaze result is well measured, but several limits matter for how far to generalize it.

Sample size. Each cell is three seeds on 50 tasks. At that size, a standard error of about 3.5 points is plausible, and the 55-point AntMaze gap is far outside it. Differences of a few points elsewhere, such as the 3-level versus 4-level Cube scores, are not.

Training budget is not matched. Training is epoch-matched across depths, so deeper models take fewer gradient steps, and training FLOPs are not matched (full text). This cuts both ways. It may understate deep models, and it makes a clean "more levels is better" reading impossible.

Deep models starve on short episodes. Clips must lie inside one episode. Push-T episodes are about 125 steps, so the 3-level model sees 58% of LeWM's clips per epoch and the 4-level model sees 14%. The authors name this as the likely cause of the collapse. They did not test it directly.

Mazes are the best case. The authors tie the benefit to a frequency gap between fast and slow entities. FourRoom, AntMaze and Humanoid show gaps of 3.1 to 12.7x and strong selective abstraction. Push-T, Cube and DROID show gaps of 1.2 to 1.6x and little. A maze with a jittery ant body and a slow-moving position is close to ideal for a hierarchy that discards fast detail.

Reproducibility. The paper and project page link code. Training seeds and planner settings are tabulated in the appendices, which is better than most. No independent replication is reported as of 13 October 2026.

Where do the gains come from?

The paper separates two levers, and this is its most useful analysis (Section 4.2 of the full text).

Goal representation. Hold the level-1 dynamics fixed and only change the space where the goal cost is computed. On AntMaze, a two-level model's native level-1 cost scores 23.3%, while the level-2 projection scores 31.3%. The best projected level-1 planner beats flat LeWM at every depth, with no upper-level planner involved. Projection costs do not clearly help on Push-T or Cube.

Temporal decomposition. Following subgoals gives a more monotone signal than a distant goal. The paper reports a negative Spearman correlation of cost against time of 0.89 to 1.00 for subgoal tracking, against 0.48 to 0.75 for HWM's level-1 goal cost (Table 10, two-level models). The authors say this alone explains HWM's gain over LeWM. H-JEPA's extra gain over HWM, mainly on AntMaze, comes from the abstract representation.

The cost-space result is the one to remember. A lot of the gain on AntMaze needs the full planner: the same trained three-level model scores 20.7 to 22.0% with a projected cost and 73.3% with the hierarchical planner.

How does it compare with related work?

What does a working researcher take from this?

• Check your frequency gap first. If the entities in your environment evolve at similar timescales, the paper's own evidence suggests a hierarchy will not help and may hurt.
• Do not stack levels blindly. Two levels was the safe choice on all four tasks. Three was best on three of four, and four never won outright.
• Watch the clip count. If your episodes are short, deeper levels lose training data. Count clips per epoch per level before blaming the architecture.
• Use cost-space projection as a cheap diagnostic. Evaluating the goal cost in an upper-level latent space needs no upper-level planner, and it tells you whether abstraction is helping.
• Real-robot video needs an action signal. On DROID, performance is zero without an inverse-dynamics loss, which the paper calls slow-feature collapse. The authors note this requires action labels.

What did the DROID experiment show?

On DROID (74,530 episodes, re-encoded by the authors), a two-level H-JEPA with inverse-dynamics supervision improved offline planning fidelity at lower planner compute, according to the authors. The metric is Frechet fidelity on the 3D end-effector path.

The caveats are the authors' own. It uses 16 held-out pick-and-place clips, 36-step open-loop plans scored offline, and no closed-loop success. The wrist camera is unused. Treat this as a feasibility signal.

What's still unclear?

• Does it transfer to real control? Every success rate is simulated. Closed-loop physical-robot control is named as the next step.
• Can deep hierarchies be trained on short data? The Push-T collapse is attributed to data scarcity, but that is a hypothesis.
• Is the hierarchy scalable beyond stride 2? The stride is fixed at 2 (3 on DROID).
• What happens without action labels? The DROID result needs an inverse-dynamics term, which needs action labels.
• How much of the gap is the baseline? The 18% flat score is low. The comparison with HWM, which reaches roughly half H-JEPA's AntMaze success, is the fairer test of the new idea.
• What does independent replication show? None is reported as of 13 October 2026.

For context on the people involved, 36Kr notes that AMI is LeCun's world-model company, and Michael Rabbat leads its world-model work. That is background, not evidence for or against the results.