World Models — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

1. What Is a World Model?
2. Classical World Models
3. Latent Dynamics: The Core Architecture
4. Foundation Video World Models (the 2024–2026 wave)
5. Architectures for Video World Models
6. Tokenizers for World Models
7. Action Conditioning
8. Long-Horizon Consistency
9. Physical Commonsense
10. World Models for Autonomous Driving
11. World Models for Robotics
12. Playable World Models / Game Generation
13. NVIDIA Cosmos Platform (Detailed)
14. Evaluation
15. Training Data and Curation
16. Open Challenges and Frontier 2026
17. Use as RL Substrate (Imagination)
18. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "Which World Model?"
• Appendix C: Year-by-Year Milestones

1. What Is a World Model?

1.1 Working definition

A world model is a generative model of how an environment evolves over time, conditioned on past observations and (optionally) actions. It can predict the next observation, or roll out arbitrarily far into the future.

\[p_\theta\!\left(\mathbf{o}_{t+1:t+H} \mid \mathbf{o}_{1:t}, \mathbf{a}_{1:t+H-1}\right).\]

1.2 The two threads

1. Latent dynamics for control (Schmidhuber's "dream" lineage; Ha & Schmidhuber 2018; Dreamer V1–V3): a learned model used as a substrate for RL. Compact latent; small recurrent / Transformer dynamics.
2. Video-foundation world models (2024–26: Sora / Veo / Cosmos / GAIA / Genie): high-fidelity video generation conditioned on actions or text, used as a learned simulator for robotics, AV, and games.

1.3 Why world models?

• Sample efficiency: train RL inside the model (imagination); fewer real interactions.
• Planning: search / MCTS over imagined rollouts.
• Closed-loop simulation: re-render real-world data under counterfactual actions (AV).
• Embodied pretraining: learn priors from video without explicit reward.
• Generative content: playable game worlds, virtual environments.
• Counterfactual reasoning: "what would have happened if. . . "

1.4 World model vs video generation

Video generation: text \(\to\) video clip. World model: state + action \(\to\) next state(s). The difference is action conditioning + the ability to roll out indefinitely under controlled inputs.

Key

A world model is a video generator with action conditioning and persistence, used as a learned simulator. By 2025 the video-gen frontier (Sora 2, Veo 3, Kling 2, Cosmos) is converging with world models.

1.5 Quality dimensions

• Fidelity: visual realism per frame.
• Temporal consistency: identity, geometry, lighting hold across time.
• Action faithfulness: actions actually steer outcomes.
• Physical commonsense: gravity, collision, occlusion, object permanence.
• Long-horizon stability: \(\sim 1\) minute clips and beyond.
• Controllability: editable, prompt-able, intervenable.
• Speed / playability: real-time interaction (Genie / Oasis frontier).

2. Classical World Models

2.1 Dyna-Q (Sutton 1990)

Mix real experience with simulated rollouts from a learned dynamics model. Update Q both ways. The original "model-based + model-free hybrid."

2.2 PILCO

Gaussian Process dynamics + analytic policy gradient via moment matching. Extremely sample-efficient on low-dim continuous control. Doesn't scale to high-dim observations.

2.3 PETS (Probabilistic Ensembles + Trajectory Sampling)

Ensemble of NNs models the dynamics; CEM (Cross-Entropy Method) plans over sampled rollouts. State-of-the-art for many MuJoCo tasks circa 2018.

2.4 Black-box vs analytic

Black-box: NN that maps \((s, a) \to s'\); no analytical gradient through dynamics. Analytic: differentiable simulator (physics engine, learned). Latter enables direct policy gradient through dynamics.

2.5 Dreamer V1 (Hafner et al. 2020)

First widely successful latent-dynamics RL world model:

• RSSM (Recurrent State-Space Model): combines deterministic + stochastic latent.
• Train via reconstruction + KL.
• Learn actor + critic in imagination.

2.6 Dreamer V2

Categorical latents (\(32 \times 32\) one-hot per stochastic state); KL balancing trick; matches model-free on Atari at 200M frames.

2.7 Dreamer V3 (2023)

Same hyperparameters across 150+ tasks (Atari, DeepMind Control, Crafter, Minecraft):

• Symlog reward / value targets: \(\mathrm{symlog}(x) = \mathrm{sgn}(x)\log(|x| + 1)\).
• Two-hot distributional heads.
• Free-bits in KL.
• Imagined horizon \(\sim 15\text{–}16\) steps.

Notable: first to collect diamonds in Minecraft from scratch via RL.

2.8 TD-MPC, TD-MPC2 (Hansen et al.)

Latent dynamics + sampling-based MPC at planning. Strong on continuous control benchmarks; robust across embodiments.

2.9 IRIS (Imagination with Auto-Regression over Inner Speech, Micheli et al.)

Tokenizer (VQ-VAE) + Transformer dynamics. Discrete tokens make the world model into an autoregressive sequence model. Strong on Atari at low data.

2.10 DIAMOND (Diffusion for Atari)

Diffusion world model for Atari at low data. Showed diffusion can replace VAE+RNN as the dynamics class.

Foundation for later video-diffusion world models.

2.11 Comparison table

3. Latent Dynamics: The Core Architecture

3.1 The encoder-dynamics-decoder pattern

1. Encoder: \(o_t \to z_t\) (image \(\to\) latent).
2. Dynamics model: \((z_t, a_t) \to z_{t+1}\) (latent transition).
3. Decoder: \(z_t \to \hat{o}_t\) (reconstruction; for training).
4. Reward head: \(z_t \to r_t\).
5. Continue head: \(z_t \to \mathbf{1}[\text{not done}]\).

3.2 Recurrent State-Space Model (RSSM)

Dreamer's core. Combines deterministic GRU + stochastic Gaussian / categorical:

\[h_t = f_\phi(h_{t-1}, z_{t-1}, a_{t-1}), \quad z_t \sim q_\phi(z_t \mid h_t, o_t), \quad \hat{z}_t \sim p_\phi(\hat{z}_t \mid h_t).\]

\(h_t\): deterministic (recurrent state). \(z_t\): stochastic (sampled). Posterior \(q\) uses observation; prior \(\hat{z}_t\) used at imagination time.

3.3 Loss

\[\mathcal{L} = \mathbb{E}_q\Big[\; \underbrace{-\log p_\phi(o_t \mid h_t, z_t)}_{\text{recon}} \; \underbrace{-\log p_\phi(r_t \mid h_t, z_t)}_{\text{reward}} \; + \cdots \Big].\]

Image / reward / continue reconstruction + KL.

3.4 Symlog / two-hot (Dreamer V3)

Symlog for reward / value targets:

\[\mathrm{symlog}(x) = \mathrm{sgn}(x)\log(|x| + 1), \quad \mathrm{symexp}(x) = \mathrm{sgn}(x)\big(\exp(|x|) - 1\big).\]

Two-hot: regress to a categorical distribution over \(\sim 256\) bins; recover scalar by expectation. Stabilizes across reward scales (Atari \(\sim 10^4\) vs DMC \(\sim 1\)).

3.5 Imagined rollouts

At policy training time, roll out the prior \(p_\phi\) for \(H\) steps (no observation); train actor + critic on imagined trajectories with REINFORCE + value baseline.

• Standard horizon: \(H = 15\text{–}16\) steps.
• Actor: maximize \(\lambda\)-return.
• Critic: regress to \(\lambda\)-return.

3.6 Why latent dynamics works

• Compresses high-dim observations to compact latent.
• Predicts in low-dim space (cheap rollouts).
• Imagination \(\to\) orders of magnitude more training data than real interaction.
• Decoder + reconstruction loss provides auxiliary signal.

3.7 Failure modes

• Imagination drift: errors compound over rollout horizon.
• Reward model overfit: actor exploits inaccurate reward predictions.
• Latent collapse: encoder ignores observation.
• Exploration: world model has no prior over unseen regions.

4. Foundation Video World Models (the 2024–2026 wave)

4.1 The shift

Video diffusion / autoregressive models trained on internet-scale video (and increasingly action-conditioned data) double as world models. Same generative model that makes Sora-class video clips can be conditioned on actions to predict next frames.

4.2 Sora / Sora 2 (OpenAI)

• Spacetime patch tokenization.
• DiT backbone.
• Re-captioning of training videos.
• Long clips (\(\ge 60\text{s}\) in Sora 2).
• OpenAI markets it explicitly as "world simulator."
• Closed; details limited.

4.3 Veo 2 / Veo 3 (Google DeepMind)

• HD long clips.
• Veo 3 adds audio.
• Camera-control conditioning.
• Closed.

4.4 Kling 1.6 / 2 (Kuaishou)

Frontier closed video model; Asia-led. Strong on photoreal motion + identity preservation.

4.5 Cosmos (NVIDIA 2025)

Cosmos World Foundation Models: explicitly positioned as world models for robotics + AV.

• Cosmos Tokenizer: continuous (CV) and discrete (DV) variants.
• Cosmos Predict (Diffusion): video diffusion.
• Cosmos Predict (Autoregressive): discrete-token AR.
• Cosmos Transfer: learned image-to-image transfer (e.g., sim-to-real).
• Cosmos Reason: VLM with physical commonsense reasoning.
• Open weights for many; commercial-grade quality.

4.6 GAIA-1 / GAIA-2 (Wayve)

• Driving-specific video diffusion.
• Conditioned on past frames + actions (steering, throttle).
• Closed-loop AV simulation: replay logged driving with counterfactual actions.
• GAIA-2: scaled up.

4.7 Genie family (DeepMind)

• Genie 1 (2024): playable 2D worlds from internet game footage; latent action discovery.
• Genie 2 (Dec 2024): 3D playable worlds from a single image prompt; up to 1 minute interactive.
• Genie 3: rumored / in development; longer playability + multi-genre.

4.8 Vista (Wayve / others)

Driving / scene-level video synthesis with controllability.

4.9 World Labs (Fei-Fei Li, 2024 startup)

3D world generation from text / image; positioned as "Large World Models." Closed previews.

4.10 Lineup table

5. Architectures for Video World Models

5.1 The dominant pattern (2025–26)

1. Causal 3D VAE encoder: video \(\to\) spatiotemporal latent grid (e.g., \(T/4 \times H/8 \times W/8\)).
2. Spatiotemporal MM-DiT: joint diffusion over latent + condition (text, action, camera).
3. Causal 3D VAE decoder: latent \(\to\) video.

5.2 Diffusion vs autoregressive

Diffusion (Sora, Veo, Cosmos Predict, GAIA):

• High visual quality.
• Parallel generation of all frames (per denoising step).
• Hard to extend indefinitely; chunked autoregression.

Autoregressive (Cosmos Predict-AR, GameNGen, Oasis):

• Naturally indefinite-horizon.
• One frame at a time; latency-friendly for interaction.
• Lower per-frame quality at same compute.
• Compatible with discrete-token tokenizers (LFQ, FSQ).

5.3 Hybrid: chunked AR + diffusion within chunks

• Generate one chunk via diffusion.
• Condition next chunk on last frames + actions.
• Iterate indefinitely.

Used in Sora-style long-clip generation.

5.4 Latent action models (Genie)

Discover discrete latent actions \(\tilde{a}_t\) from unlabeled video:

• Encoder: \((o_t, o_{t+1}) \to a_t^{\text{latent}}\) (discrete token).
• Decoder: \((o_t, a_t^{\text{latent}}) \to o_{t+1}\).
• Train end-to-end on raw video.

At inference, user controls via the discovered action vocabulary \(\to\) playable.

5.5 Camera-control conditioning

• Encode camera trajectory (6-DoF or extrinsic matrices) as condition.
• Plücker coordinates per pixel as conditioning map (CameraCtrl, MotionCtrl).
• 3D-aware: use camera pose to define world coordinates.

5.6 Key-frame conditioning

• Provide first / last / sparse key-frames.
• Generate in-betweens.
• Useful for animation, video editing, world-model rollouts with anchors.

5.7 Scene memory / persistence

Open challenge: how to remember things outside current view.

• Genie 2: explicit recurrent state; "memory" across rollout.
• Memory token banks (Sora-style long clips).
• Cached latent grid.

6. Tokenizers for World Models

6.1 Why tokenizers matter

World models live or die by their tokenizer. Bad tokenizer \(\to\) blurry rollouts, poor temporal coherence, wasted compute.

6.2 Causal 3D VAE

3D conv encoder with causal temporal padding (current frame depends only on past). Standard \(4\times\) temporal \(\times\ 8\times\) spatial compression. Used in Sora, Open-Sora, CogVideoX, Cosmos.

6.3 MAGVIT-v2 (LFQ)

Lookup-Free Quantization: project latent to dim \(L\), sign-quantize \(\to\) vocab \(2^L\). No codebook collapse. Joint image + video training. Used in Genie, Emu3, several open AR video models.

6.4 FSQ (Finite Scalar Quantization)

Per-dim rounding to a small set; vocab \(\prod_i (2K_i + 1)\). Simpler than LFQ; competitive.

6.5 Cosmos Tokenizer

NVIDIA's joint image + video tokenizer.

• Continuous variant (CV): for diffusion.
• Discrete variant (DV): for AR with FSQ.
• Causal 3D structure.
• Up to 8K resolution.

6.6 Token-budget math (per-second video)

At 24 fps, 256×256, 4× temporal × 8× spatial:

\[\text{tokens/sec} = 24 \cdot 256^2 / (8^2 \cdot 4) = 6144.\]

A 1-min clip \(\sim\) 370k tokens. Action conditioning adds a small number of tokens per step.

6.7 Compression matters for AR rollouts

AR world models pay \(O(N^2)\) attention over the cache. Aggressive temporal+spatial compression keeps long-horizon rollout feasible.

7. Action Conditioning

7.1 The conditioning problem

World models for control must respond to actions. Approaches:

7.2 Discrete action tokens

• Reserve vocab range for actions; mix into AR sequence.
• Used in IRIS, Cosmos-AR, Genie.
• Easy to integrate with text + video tokens.

7.3 Continuous action conditioning

• Embed continuous actions via small MLP.
• Inject as conditioning (cross-attention or adaLN).
• Used in Cosmos Predict, GAIA-1.
• Action as a vector per timestep.

7.4 Flow-matching action heads

For control settings (\(\pi_0\), GR00T): predict action distribution via FM. Combined with world model, supports closed-loop simulation + control.

7.5 Camera-trajectory conditioning

• Camera intrinsics + extrinsics per frame.
• Plücker coordinates per pixel.
• Or text-described motion ("zoom in," "pan left").
• Used in CameraCtrl, MotionCtrl, ViewCrafter.

7.6 Latent action discovery (Genie)

Unsupervised: encoder maps \((o_t, o_{t+1})\) to a discrete latent action; world model conditioned on it. At inference, user picks from discovered actions \(\to\) effectively playable. No action labels needed during training.

7.7 Multi-modal actions

• Text prompt + action vector.
• Reference image + camera path.
• Multi-channel control (text for content, action for dynamics).

7.8 Action faithfulness

Critical metric: do actions actually steer outcomes? Quantify via:

• Action-prediction inversion (predict action from rollout).
• Counterfactual divergence (different actions \(\to\) different rollouts).
• Closed-loop downstream success.

8. Long-Horizon Consistency

8.1 The challenge

Real-world coherence over \(\ge 10\) seconds: object permanence, identity, geometry, lighting, occlusion handling.

8.2 Approaches

• Recurrent latent state: explicit memory across timesteps (Dreamer, Genie 2).
• Anchor-frame conditioning: provide reference frames; generate in-betweens.
• Long-context transformers: ring attention over many frames.
• Memory token banks: store summary tokens of past frames.
• 3D representation handoff: maintain explicit 3D scene + render per frame (early but emerging).

8.3 Chunked autoregression

• Generate clip of length \(K\) via diffusion.
• Condition next clip on last \(k < K\) frames + actions.
• Repeat indefinitely.
• Trade-off: compounding error vs cost.

8.4 Identity preservation

Subjects (people, vehicles) drift in long rollouts. Solutions:

• Reference-image conditioning per subject.
• Identity-preserving losses during training.
• Periodic re-anchoring.

8.5 Scene-level persistence

Things outside current view drift / vanish.

• Maintain explicit scene representation (3DGS, voxel grid).
• Render as one input branch; world model fills in details.
• Genie 2's emergent persistence is impressive but not perfect.

8.6 Practical horizons (2026)

• General video (Sora 2 / Veo 3): 1–2 minutes coherent.
• Driving (GAIA-2): tens of seconds.
• Playable (Genie 2 / Oasis): up to \(\sim 1\) minute.
• Robotics (Cosmos): seconds for high-frequency control.

9. Physical Commonsense

9.1 What "physics" means here

Object permanence, gravity, collision, occlusion, conservation of mass / energy, fluid dynamics, lighting consistency. Not analytical PDEs; just the soft physical priors humans expect.

9.2 Failure modes (Sora-class)

• Objects appear / disappear arbitrarily.
• Hands / limbs morph through objects.
• Liquids ignore gravity / containers.
• Shadows inconsistent with lighting.
• Mass conservation violated.
• Multi-agent interactions non-physical.

9.3 Why physics is hard

• Generative models match training distribution, not physical laws.
• Few training samples densely capture rare physical scenarios.
• Long-horizon compounding: small errors \(\to\) catastrophic.
• Multi-object dynamics combinatorially complex.

9.4 Improving physical fidelity

• Train on physically-accurate simulation data.
• Add physics-aware losses.
• Hybrid: classical physics simulator + neural visual rendering.
• Constrained generation (Cosmos Transfer-style consistency).
• Auxiliary VLM evaluator (Cosmos-Reason) for filtering bad rollouts.

9.5 Cosmos Reason

NVIDIA's VLM trained for physical commonsense reasoning over video. Used to:

• Score rollouts for physical validity.
• Filter training data for cleaner physical signal.
• Reward signal for RL training of world models.

9.6 Hybrid physics-neural systems (frontier)

• PhysGaussian: 3DGS + continuum mechanics.
• GenSim: classical sim with neural visuals.
• Differentiable physics + NeRF / 3DGS.

10. World Models for Autonomous Driving

10.1 Why AV needs world models

• Closed-loop simulation: re-test policy in counterfactual scenarios.
• Long-tail edge case generation.
• Reduce real-world miles for testing.
• Train policy in imagination (sample efficiency).

10.2 GAIA-1 / GAIA-2 (Wayve)

Driving-specific video diffusion. Conditioned on past frames + actions (throttle, brake, steering). Closed-loop AV simulator; replay logged driving with counterfactuals.

10.3 DriveDreamer / DriveDreamer-2

Open AV world models; text-controlled scenario generation.

10.4 Vista, MagicDrive3D, EmerNeRF

Various scene-level synthesis approaches.

10.5 OccWorld / DriveWorld

3D-occupancy-grid based world models; predict per-voxel occupancy + semantics over time.

10.6 Wayve LINGO

Vision-language-action with reasoning over driving scenarios.

10.7 Closed-loop AV simulation pipeline

1. Capture real driving log.
2. Reconstruct scene (NeuRAD / EmerNeRF / Cosmos Transfer).
3. Replay with perturbed actions.
4. Render via world model.
5. Evaluate policy.

10.8 Neural reconstruction backbones for AV

StreetGaussians, EmerNeRF, S-NeRF, NeuRAD, OmniRe, UniSim: reconstruct logged driving scenes for re-rendering. Combined with world models for full closed-loop.

11. World Models for Robotics

11.1 Why robotics needs world models

• Sim-to-real bridging: visually realistic sim with learned priors.
• Imagined rollouts for sample-efficient RL.
• Plan over predicted trajectories.
• Foundation pretraining from passive video.

11.2 1X World Model

1X (humanoid robotics startup) released a world model trained on humanoid robot footage. Conditioning on past frames + commanded actions; predicts visual outcomes.

11.3 GR00T World Model (NVIDIA)

Cosmos-derived; targets humanoid robotics.

• Action-conditioned video.
• Sim-to-real transfer.
• Combined with GR00T VLA for closed-loop.

11.4 DayDreamer (Wu et al.)

Train Dreamer-style world model on real robot data; deploy for offline planning + RL fine-tune.

11.5 Pretrained passive video as embodied prior

V-JEPA 2, Cosmos Predict: pretrained on internet video; fine-tuned on robot data for control. Captures motion priors without explicit action labels.

11.6 UniSim, RoboGen, SimplerEnv

Simulator-style world models for evaluating manipulation policies.

11.7 Diffusion world models in policy learning

DPPO and related: train policy alongside diffusion-based world model; gradient through sampler.

11.8 Embodied AI roadmap (2026 view)

1. Pretrain world model on internet video.
2. Fine-tune on robot data with action conditioning.
3. Train VLA inside world model (imagination).
4. Deploy on robot; collect more data; iterate.

12. Playable World Models / Game Generation

12.1 The Genie family (DeepMind)

• Genie 1 (2024): 2D platformer worlds from internet videos. Latent action discovery; user picks discrete actions to control.
• Genie 2 (Dec 2024): 3D worlds from a single image; up to \(\sim 1\) minute interactive; emergent object permanence + character behaviors.
• Genie 3 (rumored 2026): longer playability; multi-genre.

12.2 Oasis (Decart 2024)

Open playable Minecraft world model. 20 Hz interactive on a single GPU; trained on Minecraft screen + control footage. Demonstrates real-time playability.

12.3 GameNGen (Google 2024)

Diffusion model trained on DOOM gameplay; runs DOOM-like at 20 fps via diffusion.

12.4 DIAMOND-CS (Iglesias et al. 2024)

Diffusion world model for CS:GO; playable. Showed diffusion world models scale to FPS-style real-time.

12.5 WHAM (Microsoft Bleeding Edge)

Game-specific world model trained on Bleeding Edge gameplay; supports persistent agent rollouts.

12.6 World Labs (Fei-Fei Li)

Startup focused on "Large World Models" for general 3D world generation. Closed previews of single-image \(\to\) explorable 3D world.

12.7 Trade-offs for playable worlds

• Frame rate vs quality.
• Action latency.
• World persistence on rollback / re-entry.
• Cost per concurrent player.

12.8 Use cases

• Procedural game generation.
• Custom game / level creation from prompt or sketch.
• Pretraining substrate for game-playing agents.
• Educational simulators.

13. NVIDIA Cosmos Platform (Detailed)

13.1 The components

• Cosmos Tokenizer: continuous + discrete; image + video; up to 8K.
• Cosmos Predict (Diffusion): video diffusion world model.
• Cosmos Predict (Autoregressive): discrete-token AR.
• Cosmos Transfer: structured noise / domain transfer (sim-to-real).
• Cosmos Reason: physical-commonsense VLM evaluator.
• Cosmos Curator: data curation for world-model training.

13.2 Why a platform vs single model?

World model serving has many sub-tasks: tokenize, predict, transfer \((\text{sim} \to \text{real})\), reason about plausibility, curate data. NVIDIA bundled them so robotics + AV teams can compose.

13.3 Open weights

Several Cosmos models open under permissive licenses. Drove huge adoption among open robotics + AV community.

13.4 Use patterns

1. Robotics: pretrain VLA on Cosmos-generated counterfactual rollouts.
2. AV: replay log \(\to\) Cosmos Transfer (clean sim-to-real) \(\to\) test policy.
3. General: video gen \(\to\) Cosmos Reason filter \(\to\) curated dataset.

13.5 Integration with Isaac Lab / Sim

Combine analytical physics (Isaac) with neural visuals (Cosmos). Best of both: physics correctness + photorealism.

14. Evaluation

14.1 Per-frame quality

• PSNR, SSIM, LPIPS.
• FID for distributional realism.

14.2 Video-level metrics

• FVD (Fréchet Video Distance): Inception-3D feature distance.
• VBench / VBench-2: 16+ axes (motion smoothness, dynamic degree, subject consistency, scene transition, object class, temporal flicker).
• VideoScore: learned video reward.
• T2V-CompBench, EvalCrafter: compositional / multi-aspect.

14.3 Action-faithfulness

• Counterfactual divergence: rollouts under different actions should differ.
• Action prediction: can we recover the action from the rollout?
• Downstream policy success when used as RL substrate.

14.4 Physical commonsense

• Cosmos-Reason scoring.
• Physics-aware benchmarks (PhyGenBench).
• Object-permanence tests (synthetic).
• Multi-agent interaction tests.

14.5 Closed-loop benchmarks (AV / robotics)

• NeuRAD-style scene reconstruction + replay.
• Bench2Drive, NAVSIM, DriveArena.
• LIBERO, RoboCasa, SimplerEnv (manipulation).
• Policy success rate as the ground truth.

14.6 Long-horizon stability

• Identity preservation over time.
• Scene drift quantification (re-projection error).
• Subjective ratings on long clips.

14.7 Human eval

Side-by-side preferences. Despite all the metrics, human preference remains the ground truth especially for "world-modeling" qualities.

15. Training Data and Curation

15.1 Internet-scale video

• YouTube, Vimeo, public TV.
• Filtering: aesthetic, motion, OCR for text overlay.
• Re-captioning with VLM.
• Deduplication.

15.2 Action-labeled data

• Game footage with input logs.
• Driving footage with CAN-bus actions.
• Robot demonstrations with control signals.
• Inferred actions via VLM (cheap but noisy).

15.3 Synthetic data

• Game engine renders (Unity, Unreal).
• Physics simulator outputs (MuJoCo, Isaac).
• Procedural generation.
• Used to teach physics priors that internet video lacks.

15.4 Self-supervised action discovery

For unlabeled video: train a latent action model (Genie-style); use discovered actions for downstream.

15.5 Curated robot datasets

Open-X-Embodiment: 1.4M trajectories across 22 embodiments. Foundation for VLA + world model training.

15.6 AV datasets

nuScenes, Waymo Open, Argoverse 2, KITTI-360, Lyft Level 5: standard. Commercial: Tesla / Wayve / Cruise have orders of magnitude more proprietary.

16. Open Challenges and Frontier 2026

16.1 Long-horizon stability

\(\sim 1\) minute is the current ceiling for general video (Sora 2 / Veo 3). Extending to 10+ minutes coherent is open.

16.2 Real-time playability

Genie 2 and Oasis are interactive but limited. Real-time at full quality (60 fps, 4K) on consumer hardware is years away.

16.3 Multi-agent dynamics

World models struggle with multiple interacting agents (cars merging, hands negotiating). Compositional / structured representations may be needed.

16.4 Physical correctness

Soft commonsense \(\to\) explicit physics. Hybrid systems (PhysGaussian, GenSim) emerging. Cosmos-Reason as evaluator helps.

16.5 Editability and intervention

Make-this-change midway through a rollout. Currently brittle; reset-and-re-run is the pattern.

16.6 Causal correctness

World models that capture causal structure, not just correlations. Open research; intersects with structural causal models.

16.7 Evaluation rigor

Many metrics; most game-able. Closed-loop downstream success is the only bulletproof measure but expensive.

16.8 Open research directions (2026)

• Hybrid neural + analytical physics.
• Better latent action discovery.
• Multi-resolution scene memory.
• Cross-embodiment world models.
• Long-form (multi-minute) coherent generation.
• Real-time on edge hardware.
• Causal interpretability.

17. Use as RL Substrate (Imagination)

17.1 Train policy in imagination

• Roll out world model from current state.
• Collect imagined trajectories.
• Train actor + critic on imagined returns.
• Periodically refresh world model with real interactions.

17.2 Dreamer V3 imagination loop

1. Encode real observations to latent.
2. Roll out RSSM for \(H = 16\) steps.
3. Predict reward + continue per step.
4. Compute \(\lambda\)-return.
5. Update actor (REINFORCE) and critic (regression).

17.3 Planning in world models

• MPC: sample \(N\) action sequences; rollout in world model; pick best.
• MCTS: tree search over world-model-predicted states.
• TD-MPC2: model-predictive control with learned latent dynamics.
• AlphaZero-in-world-model patterns.

17.4 Counterfactual for AV / robotics

• Replay logged scenario.
• Inject counterfactual action.
• Generate plausible alternative outcome.
• Use for safety analysis, edge-case testing.

17.5 Pretraining VLAs in world models

• Pretrain on passive video (no actions).
• Fine-tune with action conditioning.
• Distill into VLA + small world model for deployment.
• Cosmos + GR00T pattern.

17.6 Limitations

• Imagination drift: errors compound; train policies that are robust to model errors.
• Reward model quality: actor exploits inaccurate reward predictions.
• Coverage: world model has no information about regions not seen in training.
• Sim-to-real gap when world model is the simulator.

18. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. World model = generative model of environment dynamics conditioned on actions.
2. Two threads: latent dynamics for control vs foundation video world models.
3. Dyna-Q: original mix of model-based + model-free.
4. PILCO: GP dynamics; sample-efficient on low-dim.
5. PETS: NN ensembles + CEM planning.
6. Dreamer V3: RSSM + symlog + two-hot; same hyperparameters across 150+ tasks.
7. RSSM: deterministic GRU + stochastic latent.
8. Symlog: \(\mathrm{sgn}(x)\log(|x| + 1)\) for cross-task robustness.
9. TD-MPC2: latent dynamics + MPC; strong continuous control.
10. IRIS: tokenizer + Transformer dynamics; Atari at low data.
11. DIAMOND: diffusion world model (Atari, CS:GO).
12. Sora: spacetime patches + DiT; OpenAI's "world simulator".
13. Veo 3: + audio; closed.
14. Cosmos: NVIDIA platform (Tokenizer / Predict / Transfer / Reason / Curator).
15. GAIA-1/2: Wayve's AV world model.
16. Genie 1/2: latent action discovery; playable worlds.
17. Oasis: open real-time Minecraft world model.
18. GameNGen: DOOM via diffusion at 20 fps.
19. Causal 3D VAE for video tokenization.
20. LFQ (MAGVIT-v2) for discrete video tokens.
21. Latent action models discover actions from unlabeled video.
22. Camera control via Plücker coordinates / extrinsics conditioning.
23. Long-horizon: chunked AR + anchor frames.
24. Physical commonsense is the open challenge; Cosmos Reason as evaluator.
25. Closed-loop downstream success is the only bulletproof eval.

Appendix B: Decision Tree — "Which World Model?"

1. Classical RL benchmark, sample efficiency? \(\to\) Dreamer V3 / TD-MPC2 / IRIS.
2. Need diffusion-quality but Atari-scale data? \(\to\) DIAMOND.
3. Driving / AV closed-loop simulation? \(\to\) GAIA-2 / Cosmos + StreetGaussians.
4. Humanoid / general robotics? \(\to\) Cosmos + Isaac Lab + GR00T or 1X World Model.
5. General video as world simulator? \(\to\) Sora 2 / Veo 3 / Cosmos Predict (open).
6. Want action-conditioned open weights? \(\to\) Cosmos Predict (NVIDIA) or DriveDreamer-2.
7. Playable / game generation? \(\to\) Genie 2 (closed) or Oasis (open) or GameNGen.
8. Single-image \(\to\) explorable 3D world? \(\to\) World Labs (closed) or Genie 2.
9. Internet-video pretraining for embodied? \(\to\) V-JEPA 2 / Cosmos Predict.
10. Need physical commonsense filter / evaluator? \(\to\) Cosmos Reason VLM.

Appendix C: Year-by-Year Milestones

• 1990: Sutton's Dyna; Schmidhuber early world-model proposals.
• 2011: PILCO (Deisenroth & Rasmussen) — GP world model.
• 2018: PETS; Ha & Schmidhuber's "World Models" paper (VAE+RNN+controller).
• 2019–2020: Dreamer V1/V2; PlaNet; Plan2Explore.
• 2022: TD-MPC; Iso-Dream; SimMIM-style passive video pretraining (V-JEPA precursor).
• 2023: Dreamer V3; IRIS; DIAMOND; GAIA-1 (Wayve); MagicDrive; DriveDreamer.
• Feb 2024: Sora announced (OpenAI); positioned as "world simulator."
• 2024: Veo 1; Genie 1 (DeepMind, 2D playable); Cosmos announced (NVIDIA); GameNGen (DOOM); Oasis (Decart, playable Minecraft); MovieGen; Kling 1.6; DIAMOND-CS (CS:GO).
• Dec 2024: Genie 2 (3D playable, 1 min interactive).
• 2025: Sora 2, Veo 3 (with audio), Kling 2; Cosmos World Foundation Models released by NVIDIA (open + commercial); GAIA-2; 1X World Model; World Labs founded; Genie 3 rumored; Wayve LINGO.
• 2026: World models as standard substrate for robotics + AV; multi-minute coherent generation; playable worlds advancing toward production game-engine replacement; Cosmos / GR00T as platform-style adoption; physical commonsense + Cosmos-Reason as evaluation standard.