World Model
A world model is an AI system that learns how its environment evolves and how actions change it, so agents can imagine outcomes and plan before acting.
A world model is a learned model of how an environment changes over time and in response to actions. Given the current state or a history of observations, plus an action, it predicts what happens next: the next observation, a compact internal state, a reward, or a sequence of future steps. An agent with a world model can imagine the results of actions before taking them, plan by comparing imagined futures, and learn from simulated experience instead of only real trial and error. The idea comes from model-based control and reinforcement learning, and David Ha and Jürgen Schmidhuber's 2018 paper World Models popularized the term for neural networks that learn a compressed simulator of a game from pixels and train a policy inside its "dream." Today the term covers the internal dynamics models of agents such as Dreamer and MuZero, interactive video generators such as Google DeepMind's Genie 3, and world foundation models that generate video for training physical AI.
How World Models Work#
A world model usually has three parts. An encoder compresses each high-dimensional observation, such as a camera frame, into a small latent state. A dynamics model predicts the next latent state from the current one and an action, often together with the reward and whether the episode ends. An optional decoder turns latent states back into images, for training or for showing what the model imagines. Ha and Schmidhuber built these parts from a variational autoencoder, a recurrent network that predicts the next latent, and a tiny linear controller trained on the latents.
An agent uses the model in two ways. In planning, it rolls candidate action sequences through the model, scores the imagined outcomes, and executes the best one, as in model predictive control or the tree search of MuZero. This PyTorch code learns a world model of a simple cart from random transitions and plans a 20-step push sequence in imagination:
import torch
from torch import nn
torch.manual_seed(0)
# The real environment, unknown to the agent: a cart's [position, velocity] responds to a push in [-1, 1]
A, B = torch.tensor([[1.0, 0.9], [0.0, 0.9]]), torch.tensor([0.1, 0.1])
# Learn a world model that predicts how (state, action) changes the state, from 5,000 random transitions
states, actions = torch.rand(5000, 2) * 4 - 2, torch.rand(5000, 1) * 2 - 1
change = states @ A.T + actions * B - states
model = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 2))
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
for step in range(2000):
loss = ((model(torch.cat([states, actions], dim=1)) - change) ** 2).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Plan by imagination: roll 1,000 random 20-step plans through the model; also run them for real to compare
plans = torch.rand(1000, 20, 1) * 2 - 1
imagined, real = torch.zeros(1000, 2), torch.zeros(1000, 2)
for k in range(20):
imagined = imagined + model(torch.cat([imagined, plans[:, k]], dim=1)).detach()
real = real @ A.T + plans[:, k] * B
best = (imagined[:, 0] - 3).abs().argmin() # plan whose imagined final position is closest to 3
print(f"chosen plan: imagined final position {imagined[best, 0]:.2f}, real final position {real[best, 0]:.2f}")It prints chosen plan: imagined final position 3.00, real final position 2.93. The agent picked its plan without touching the real cart, and the 0.07 gap is model error that accumulated over 20 imagined steps. Small errors compound over long rollouts, which is why practical planners re-plan after every step and keep imagined horizons short.
In learning by imagination, the agent trains its policy on rollouts generated inside the model. DreamerV3, published in Nature in 2025, alternates between collecting real experience, improving the world model, and training an actor and critic purely on imagined trajectories:
With one set of hyperparameters, DreamerV3 outperformed specialized methods across more than 150 tasks and became the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula.
Types of World Models#
- Agent dynamics models: Dreamer predicts future latent states, rewards, and episode ends. MuZero predicts only what planning needs, the reward, policy, and value, and with Monte Carlo tree search it matched AlphaZero at Go, chess, and shogi without being given the rules.
- Interactive video models: Genie, an 11-billion-parameter model trained on unlabeled internet videos of 2D platformer games, learned its own latent actions and turns a single image, photo, or sketch into a controllable game world. Genie 3 generates worlds that users navigate in real time at 24 frames per second and 720p, staying consistent for a few minutes.
- World foundation models: NVIDIA's Cosmos provides open-weight models that predict future video from text, images, or past frames, intended for post-training on specific robots and vehicles.
- Predictive embedding models: A joint embedding predictive architecture (JEPA) predicts the representation of future or missing content instead of its pixels, so the model does not spend capacity on unpredictable details such as the exact pattern of moving leaves.
Object-centric world models represent a scene as a set of objects and predict how each one moves. Ultralytics YOLO26 in tracking mode turns video into this kind of state: boxes for every object, linked across frames by persistent IDs.
World Models vs. Related Concepts#
- Video generation: A video generator produces a plausible clip from a prompt. A world model predicts the consequences of actions, so it must respond correctly when the action changes. OpenAI described video generators as potential world simulators, and models such as Genie 3 that take a user's controls frame by frame sit between the two.
- Model-free reinforcement learning: Model-free agents learn a policy or value function directly from real experience without predicting how the environment changes. Model-based agents learn a world model and reuse it to plan or to generate training experience, which usually needs far fewer real interactions.
- Simulators and digital twins: Physics engines and digital twins are built by engineers from equations, CAD models, and measured parameters. A world model is learned from data, so it can capture effects that are hard to write as equations, such as how cloth folds or how pedestrians react to traffic.
- Large language models: A language model predicts the next text token. A world model predicts the next state of an environment, usually from video, sensor data, and actions.
Real-World Applications#
- Autonomous driving: Wayve's GAIA-1 generates driving video conditioned on past video, text, and ego-vehicle actions. The Waymo World Model, built on Genie 3 and announced in February 2026, generates both camera and lidar data to simulate situations the fleet has never recorded, such as a tornado or an elephant on the road.
- Games and interactive media: Interactive video models turn a text prompt or an image into an explorable world that responds to keyboard input.
- Robotics: Meta's V-JEPA 2 was pretrained on more than 1 million hours of internet video. Its action-conditioned version, V-JEPA 2-AC, was post-trained on less than 62 hours of unlabeled robot video and then picked and placed objects on Franka arms in two new labs by planning toward goal images, with no data collected from those robots or environments.










