Building Effective World-Action Models on Predictive Visual Latents
We study whether a world-action model can be built on a frozen predictive visual encoder without inheriting a complete pretrained visual generator. V-JEPA Policy learns a future predictor and action expert on that latent space, which also supports transferring future-modeling knowledge from DROID videos and instructions to robot control.
*Visual encoder + future predictor + action expert. The frozen T5 text encoder is used to precompute instruction embeddings and is excluded from this policy parameter count.
We study what a WAM needs to inherit, how its visual foundation affects robustness, and whether the same V-JEPA latent space supports acquiring and transferring future-modeling knowledge.
The future predictor and action expert are learned jointly from downstream demonstrations on frozen V-JEPA 2.1 latents. Without inheriting a complete pretrained visual generator, the 0.9B From Scratch policy reaches 97.25% on LIBERO and 50.92% on RoboCasa-GR1.
Benchmark results (Table 5)With the policy backbones, demonstrations, and training budget held fixed, V-JEPA 2.1 ViT-L leads the tested encoders on LIBERO-Plus. Its advantage over DINOv2 widens from 2.35 points on LIBERO to 12.23 points under distribution shifts.
Visual foundations and encoder scale (Table 4)The same frozen V-JEPA 2.1 encoder defines the latent inputs and future-prediction targets in DROID pretraining and downstream policy learning. A predictor pretrained without action labels transfers its future-modeling knowledge through its weights to joint training with a new action expert, raising LIBERO-Plus success from 79.25% to 91.50% at the same downstream budget.
Learning and transfer in V-JEPA latent space (Figure 2)The From Scratch design tests the sufficiency of predictive visual latents directly. The pretrained visual encoder stays frozen, while robot demonstrations jointly train a new future predictor and action expert. Their layer-wise context key–value states connect future modeling to action generation; language and proprioception condition both modules.
V-JEPA 2.1 ViT-L encodes each camera view into spatiotemporal tokens. The encoder is shared across pretraining and policy learning and never receives downstream gradients.
Instruction-conditioned future queries predict the next latent tubelets in one forward pass. Context states become future-informed through bidirectional interaction.
A flow-matching action expert attends to the predictor's layer-wise context key-values and produces a 32-step action chunk without rendering imagined pixels.
From Scratch jointly learns the future predictor and action expert on downstream demonstrations. Pretrained Predictor adds predictor-only pretraining on DROID before the same downstream training, with a newly initialized action expert. The visual and text encoders remain frozen in both variants.
| Benchmark | From Scratch | Pretrained Predictor |
|---|---|---|
| LIBERO | 97.25 | 98.70 |
| LIBERO-Plus | 79.25 | 91.50 |
| RoboCasa-GR1 | 50.92 | 55.58 |
Matched downstream data, protocols, and training budgets; predictor pretraining adds an upstream stage. LIBERO-Plus uses the LIBERO-trained policies without further fine-tuning.
Compare discriminative, reconstructive, video-understanding, and predictive representations under one downstream recipe. V-JEPA 2.1 ViT-L achieves the strongest LIBERO-Plus result among the tested encoders, with a larger margin under distribution shifts.
| Frozen encoder | LIBERO | LIBERO-Plus |
|---|---|---|
| DINOv2 ViT-L/14 | 94.90 | 67.02 |
| DINOv3 ViT-L/16 | 93.35 | 64.28 |
| WAN2.2 VAE | 92.60 | 46.43 |
| InternVideo3 | 93.10 | 68.62 |
| V-JEPA 2.1 ViT-L | 97.25 | 79.25 |
Upgrading V-JEPA 2 to 2.1 improves robustness at both encoder sizes, while increasing capacity gives mixed returns. V-JEPA 2.1 ViT-L achieves the strongest LIBERO-Plus result, outperforming its ViT-G counterpart under the same downstream recipe.
| Predictive encoder | Encoder size | LIBERO-Plus |
|---|---|---|
| V-JEPA 2 ViT-L | 304M | 74.15 |
| V-JEPA 2.1 ViT-L | 304M | 79.25 |
| V-JEPA 2 ViT-G | 1.01B | 76.08 |
| V-JEPA 2.1 ViT-G | 1.01B | 78.37 |
A shared V-JEPA 2.1 encoder maps current observations and future frames into the same frozen latent space in both stages. On DROID videos, instructions and observed state condition a predictor trained to forecast future latents, without action labels or an action expert. Its learned weights then initialize the downstream predictor, which supplies future-informed context to a newly initialized action expert during joint policy learning. The latent space thus supports both direct WAM learning and the acquisition and transfer of broader future-modeling knowledge.
LIBERO-Plus success rates from Table 5 and Figure 2. The extended scratch run keeps the model, dataset, and batch size fixed. Predictor pretraining adds an upstream stage, so these results do not compare equal total training compute.
A LIBERO-Spatial in-domain rollout is shown alongside seven LIBERO-Plus generalization cases. LIBERO-Plus evaluates LIBERO-trained policies under distribution shifts without further fine-tuning. These clips are qualitative examples; aggregate results appear above.
In-domainLIBERO
Pick up the black bowl between the plate and the ramekin and place it on the plate.
GeneralizationLIBERO-Plus
Grasp the darkhued rounded container situated between the flat dish used for serving food and the small glazed ceramic dish and set it down onto the flat dish used for serving food.
GeneralizationLIBERO-Plus
Pick up the black bowl between the plate and the ramekin and place it on the plate. Background textures are perturbed.
GeneralizationLIBERO-Plus
Pick up the black bowl between the plate and the ramekin and place it on the plate. The robot starts from a perturbed initial state.
GeneralizationLIBERO-Plus
Pick up the black bowl between the plate and the ramekin and place it on the plate. Camera viewpoint is perturbed.
GeneralizationLIBERO-Plus
Pick up the black bowl between the plate and the ramekin and place it on the plate. Object layout is perturbed.
GeneralizationLIBERO-Plus
Pick up the black bowl between the plate and the ramekin and place it on the plate. Sensor noise is applied to observations.
GeneralizationLIBERO-Plus
Pick up the black bowl between the plate and the ramekin and place it on the plate. Lighting conditions are perturbed.
LIBERO-Plus gains are aggregate improvements from predictor pretraining, reported in percentage points in Figure 2. They are not scores calculated from the individual clips shown here.
Read each row from left to right: an observed context latent followed by four predicted temporal latents. Rows separate the main and wrist cameras. Colors show a shared PCA projection of normalized latent tokens on the 14×14 patch grid; they are not decoded RGB frames.
| Camera view | Observed | Predicted future | |||
|---|---|---|---|---|---|
| Context | Future 1 | Future 2 | Future 3 | Future 4 | |
| Main camera |  |
 |
 |
 |
 |
| Wrist camera |  |
 |
 |
 |
 |
Scroll horizontally to follow the full prediction sequence.
On the TianJi Marvin dual-arm platform, the Pretrained Predictor improves two long-horizon bimanual tasks. Table 3 reports 20 trials per task: Table Cleanup improves from 11/20 to 15/20, and Saucer Racking from 7/20 to 17/20.
Latency and memory: Table 6 / Appendix F, action-prediction core on an RTX 4090, batch size 1, three camera views. These measurements exclude sensor I/O and motor execution; 4.66 GiB is peak PyTorch allocated memory.
Start with the LIBERO tutorial, then use the benchmark guides for LIBERO-Plus and RoboCasa-GR1. The repository includes predictor pretraining, downstream joint training, evaluation examples, and agent skills. Prepare the visual encoder weights, data, and text caches before launching a training recipe.
git clone --branch release \
https://github.com/breez3young/VJEPA-Policy.git
cd VJEPA-Policy
conda create -n vjepa-policy python=3.10 pip -y
conda activate vjepa-policy
# Install PyTorch for your CUDA driver (see Setup).
python -m pip install -e .
# Follow the training guide to prepare data and weights.From Scratch and Pretrained Predictor, with LIBERO as the tutorial example.
Training guide ↗Benchmark setup, episode counts, and result validation live alongside each example.
Choose a benchmark ↑LIBERO and RoboCasa-GR1 policies, plus the DROID-pretrained predictor, are planned releases. Researchers can train with the current recipes.
Release status ↗If you find V-JEPA Policy useful for your research, please consider citing:
Download BibTeX@article{zhang2026vjepa-policy,
title={V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents},
author={Zhang, Yang and Zhao, Jiangyuan and Fan, Chenyou and Hu, Jiayu and Yuan, Xiu and Bai, Chenjia and Li, Xiu},
journal={arXiv preprint arXiv:2609.37250},
year={2026}
}