Himanshu Pathak
WritingWorld models
Research notes · October 2026

The World
Models Today.

What does a model need to know
before an agent can act?

From a moving particle to latent imagination: how different world models represent state, predict change, and use those predictions to choose an action.

By Himanshu Pathak~33 min read · animated figures
Read the essay
One present, three action-conditioned futuresA phase portrait of a damped spring. Three trajectories share the same initial position and velocity but evolve under different constant forces.position →velocitya = -0.35a = 0a = 0.35one presentThree actions. Three counterfactual rollouts.
FIG. 00 One state, three actions. Each curve is a computed future in a spring system.

A single image of a ball leaves something unresolved: which way is it moving? Add another frame and the answer changes. Add an action, and prediction becomes a question about consequences.

That small example runs through this article. It connects state estimation to learned dynamics, then to the representations and planning methods used in current world-model research.

A map of the ideas in this article
The conceptual map
  1. 1 Learn the language Probability · linear algebra · gradients
  2. 2 Specify the decision MDPs · POMDPs · rewards · Bellman equations
  3. 3 Identify the dynamics State estimation · system identification · Dyna
  4. 4 Choose a representation Latent states · RSSMs · recurrent memory

From here, the questions branch.

Plan in task spaceMuZero · TD-MPC2Model a sequenceIRIS · TransformersPredict featuresJEPA · V-JEPAGenerate environmentsGenie · video models
Make predictions useful for decisions. Embodiment · uncertainty · causal evidence · evaluation
01 / Foundations

What a world model predicts

A world model is useful because an agent cannot try every possible future. It needs a way to represent its situation, anticipate consequences, and connect those consequences to a decision. Learning the subject therefore brings together probability, representation learning, dynamical systems, reinforcement learning, and control.

Begin with a decision problem. An agent receives an observation, selects an action, and receives another observation and a reward. A model can predict some part of that transition. A policy chooses the action. A reward or cost specifies what the decision is trying to achieve. These are related components, but they are different objects.

The useful question is “What must be predicted well enough for this decision?” For stabilizing a pendulum, angular position and velocity matter. For navigation, an occupied doorway matters. For visual scene generation, appearance matters too. An abstraction is useful only relative to what the system will do with it.

The mathematics underneath

The equations draw on a few recurring ideas. Vectors describe state, covariance describes how uncertainty is distributed, and gradients tell us how a prediction changes when its parameters change. Probability becomes essential as soon as one observation admits several possible explanations.

In deep learning, connect those ideas to an encoder, an autoregressive sequence model, a recurrent memory, and a generative objective. A variational autoencoder adds a latent-variable interpretation; attention adds a way to condition on a sequence of evidence. The prerequisite is understanding the mechanism well enough to identify the input, output, loss, and information available at prediction time.

Keep Essence of Linear Algebra, Deep Learning, and Dive into Deep Learning available as references. For a deeper probabilistic treatment, use Murphy’s Advanced Topics. You can learn a missing piece when the next experiment actually needs it.

Where reinforcement learning enters

An MDP describes states, actions, rewards, and transition probabilities. A policy maps the available decision state to an action distribution. A value function estimates future return under a policy. The Bellman relationship connects the value of a state to immediate reward and the value of possible next states.

Vπ(s)=E[r+γVπ(s′)∣s,a∼π]

Value is about an objective and a policy. Dynamics alone do not define what the agent wants.

The expectation accounts for uncertain actions or transitions. The discount γ controls how future rewards contribute in this formulation. Policy evaluation, temporal-difference learning, and actor-critic methods provide several places to use model-generated experience. Sutton and Barto is the starting reference.

Three ways a prediction can matter

A planner can search predicted futures at decision time. A learner can improve its policy using simulated experience. An evaluator can use a prediction to identify a failure, select data, or decide when to obtain new evidence. A system may combine several of these uses.

The recent Plausible → Controllable → Actionable survey offers a useful capability hierarchy: preserve relevant structure; respond appropriately to interventions; and demonstrate that prediction improves a behavior or workflow. Use it as an evaluation lens, not as a claim that every paper already satisfies all three levels.

The same five choices recur across these systems: state, prediction target, action interface, objective, and the decision being evaluated.

02 / State and belief

Observation and hidden state

A picture of a moving object tells you where it appears now. It does not necessarily tell you its velocity. Two physically different situations can produce the same observation while requiring different actions.

That is the central difficulty of partial observability. The agent has evidence about a state; it may not have the state itself. A POMDP makes this distinction explicit. Instead of pretending that the latest image contains everything, the agent can maintain a belief: a distribution over hidden states consistent with its history.

bt(s)=p(st=s∣o1:t,a1:t−1)

Observations constrain a hidden state; history can change which states remain plausible.

In the animation below, the two possible velocities have equal weight by construction. The current position agrees with both. Revealing the previous observation resolves which direction the object was moving in this noise-free example. With noisy measurements, evidence would usually reweight possibilities instead of eliminating one perfectly.

Figure 1: One position admits two velocities. An earlier observation resolves the ambiguity in this noise-free example. Two particles share the same observed position and move in opposite directions. Revealing an earlier observation rules out one direction. One observed positionearlierx = 0.5v = +0.16v = −0.16v ≈ (0.50 − 0.42) / 0.5 = +0.16
Fig. 01. One position admits two velocities. An earlier observation resolves the ambiguity in this noise-free example.

State estimation and memory

An estimator combines a prediction with new evidence. A Kalman filter does this analytically under linear-Gaussian assumptions. Extended and other nonlinear filters relax parts of that setting. A learned recurrent system may compress evidence into memory without exposing a human-readable state distribution.

A recurrent hidden vector can retain history without representing calibrated uncertainty. Its usefulness depends on which distinctions survive compression and whether they matter for the decision. The MIT state-estimation notes provide a control-oriented reference.

There is a useful separation between state estimation and system identification. Estimation asks “What state is the system in now?” Identification asks “What dynamics explain the observed transitions?” These can interact, but improving one does not automatically solve the other.

two identical images, two different actions?

A pendulum can pass through the same angle while moving clockwise or counterclockwise. If the correct force depends on angular velocity, the angle image alone is insufficient. History or a velocity measurement can distinguish the states.

This ambiguity also affects control. A policy receiving only position has to choose the same action for both velocities, even when one state needs braking and the other needs acceleration.

03 / Learning the transition

Learning dynamics from data

Start with an inspectable physical system. For a spring-like particle, a state contains position x and velocity v. An action u applies force. Stiffness k pulls toward the equilibrium; damping c reduces velocity.

dvdt=u−kx−cvdxdt=v

Here k and c are fitted from sampled states, actions, and noisy acceleration measurements.

We generate transition measurements from known coefficients k = 0.8 and c = 0.15. The learner begins with incorrect coefficients and minimizes mean squared error through gradient descent. A separate seeded dataset supplies held-out evaluation. For this example, 96 measurements with Gaussian noise σ = 0.03 supply the training data.

Both springs below begin at the same position with zero velocity. On the left, the predictor uses its initial guess; on the right, it uses the coefficients after 120 gradient steps. The outlined block follows the true system. The colored block follows the prediction. Their separation makes the rollout error visible.

Figure 2: The same six-second spring rollout before and after fitting stiffness and damping. The outlined block and dotted curve show the true system; color shows the prediction. Two animated spring systems compare predictions before and after fitting two dynamics parameters using 96 synthetic measurements. Before fittingk = 0.15 · c = 0.02position error 0.78806 s— prediction ··· true motionAfter fittingk = 0.809 · c = 0.147position error 0.01706 s— prediction ··· true motion
Fig. 02. The same six-second spring rollout before and after fitting stiffness and damping. The outlined block and dotted curve show the true system; color shows the prediction.

Where this example stops

The model class already contains the correct equation, and it only has two trainable parameters. This is a deliberately small system-identification problem. It does not reproduce the difficulty of discovering objects, learning a visual representation, or predicting contact dynamics with a neural network.

Even here, test the predictor under conditions that differ from the training data. A low error on randomly sampled transitions does not establish that a model remains accurate over a long recursively generated trajectory. In a real experiment, split whole episodes before evaluating to avoid near-duplicate neighboring frames crossing the split.

For a neural version, provide the current state and action to an MLP and predict the next state or a state change. Use a stochastic model if the transition or measurement process requires a distribution. Record the representation, normalization, loss, and dataset coverage so an error curve has an interpretable meaning.

From identification to model-based RL

Dyna connects real experience, model learning, simulated transitions, and value updates. PILCO brings probabilistic dynamics and uncertainty into policy search. PETS combines probabilistic ensembles with trajectory sampling. MBPO studies how to use model-generated experience without trusting arbitrary rollout lengths.

Together, these methods connect fitting a predictor to using its simulated experience. Their differences concern uncertainty, model bias, and how far to trust a rollout. The original sources are Dyna, PILCO, PETS, and MBPO.

Fitting minimizes acceleration error on sampled transitions. The animation shows a different test: the position reached after integrating those accelerations over six seconds.

04 / Representation and imagination

Latent state and imagination

Images are large, redundant observations. A latent model learns a representation in which predicting important changes can be easier. The challenge is to discard irrelevant detail while preserving the distinctions needed for future decisions.

The original World Models work combines observation compression, a recurrent predictive model, and a controller. PlaNet advances planning from pixels with learned latent dynamics and a recurrent state-space model. The conceptual jump is from “predict the next measurement” to “maintain a useful, history-aware predictive state.”

The two paths through an RSSM

A recurrent state-space model can combine deterministic memory h with a stochastic latent variable z. Memory carries forward information from the preceding state and action. A predictive prior estimates the current latent using that memory. When an observation is available, an observation-conditioned posterior can update the latent representation.

Predictive and observation-conditioned paths in an RSSMDeterministic memory supports a predictive prior over latent state. An observed image also informs a posterior. During imagination the system uses the predictive path without new observations.A recurrent state-space model: the two information pathsMemory hₜpast latent + past actionPredictive priorp(zₜ | hₜ)Imagined latentsample zₜ without new pixelsObservation oₜnew sensory evidencePosterior correctionq(zₜ | hₜ, oₜ)Model state (hₜ, zₜ)reward · continue · decodeDuring imagination, no new observation is available to correct the prior.Schematic RSSM information flow. Parameterizations differ across papers.
Predictive and observation-conditioned information paths. Paper-specific distributions and training objectives differ.

In symbols, the distinction is between p(zt | ht) and q(zt | ht, ot). During imagination, the model has no new real observation with which to correct the prediction. It must continue through its predictive path.

The next animation follows a scalar Gaussian update. It is an analytic filter example, not a Dreamer implementation. A precise measurement receives more weight; a noisy measurement leaves more influence with the prior.

Figure 3: A Gaussian prior, a measurement likelihood, and their resulting posterior. The curves appear in sequence; this is one analytic update, not a succession of updates. The prior distribution appears first, then an observation likelihood centered at 0.6, then the posterior centered at 0.48. PriorObservationPosteriorProbability density-101μ′ = 0.48μ = 0 → μ′ = 0.48
Fig. 03. A Gaussian prior, a measurement likelihood, and their resulting posterior. The curves appear in sequence; this is one analytic update, not a succession of updates.

The prior has mean 0 and standard deviation 0.5. The observation is 0.6 with measurement noise σ = 0.25. Combining their precisions gives a posterior mean of 0.48 and standard deviation of about 0.224. The curves appear in sequence to show the ingredients of this single update.

What the training objective must preserve

Reconstruction can encourage a latent state to retain observation information. Reward and continuation prediction can preserve signals relevant to return and episode termination. Predictive or representation losses connect observation-conditioned states to states the model can generate without new evidence. Multi-step consistency must also be investigated.

A representation can become easy to predict by discarding important distinctions. Conversely, a highly detailed representation can be expensive to model and still fail to preserve the variable a controller needs. “Compact” and “useful” are different properties.

write down a latent model’s training problem.

A simplified variational state-space objective combines observation reconstruction, optional task-signal prediction, and a penalty aligning an observation-conditioned posterior with a predictive prior. The expectation is taken over the inferred latent trajectory. The exact factorization and weighting depend on the model.

ℒ=Eq[ℒreconstruction+ℒtask]+βDKL(q(z∣h,o)‖p(z∣h))

This is a teaching schematic, not DreamerV3’s exact implementation or full sequence objective.

The reconstruction term discourages throwing away everything in the input. The predictive alignment term asks the model to produce useful latent states without new observations. Neither alone establishes useful control. Trace where gradients flow and which quantities are stopped, balanced, or clipped in the implementation you actually study.

Why the Dreamer sequence matters

Dreamer uses latent imagination to learn behavior. DreamerV2 develops discrete latent representations. DreamerV3 develops a more robust recipe across varied tasks. Dreamer 4 explores scalable world-model training and behavior learning from offline data in a different setting.

In the implementation, the interesting boundary is where observed experience ends and imagined experience begins. The posterior has access to an image; the imagined actor and critic depend on states generated without one. Tensor shapes describe the computation, but that information boundary explains what the computation is allowed to know.

why can a one-step model look good and imagination still fail?

One-step evaluation can condition on a true observed state. A recursive rollout conditions later predictions on the model’s own outputs. Those inputs can drift from the states seen in training, and there is no real observation to correct them during the imagined sequence.

05 / Planning and control

Planning with an imperfect model

A dynamics model answers what might happen. A planner needs an objective to decide which action sequence to prefer. It can simulate candidates, accumulate predicted cost, and optimize over actions.

a0:H−1*=arg mina0:H−1E[∑t=0H−1c(st,at)+Vterminal(sH)]

The terminal estimate summarizes what lies beyond a finite planning horizon.

In the following animation, the planner samples and optimizes candidate action sequences. Four force intervals define each sequence. A cross-entropy method keeps an elite set and updates the sampling mean and spread. The objective rewards approaching the goal through cost reduction and penalizes effort, unsafe boundaries, and the obstacle when that obstacle is included in the predictive model.

Across six passes, the candidate paths concentrate around lower-cost solutions. The selected sequence then runs in an environment with collision rules. When the predictive model omits the obstacle, the imagined and executed paths separate: the optimizer has found a good answer to an incomplete problem.

Both comparisons use seed 14, a 28-step horizon, and 96 candidates per pass. The only changed assumption is whether the model includes the obstacle. Different seeds and horizons can produce different outcomes; this pair illustrates model mismatch rather than estimating average performance.

Figure 4: Two optimizers use the same seed and budget. Green traces show predicted paths; dark traces show execution. The right model omits the obstacle, but the environment still enforces collisions. Side-by-side particle plans show sampled paths refining over six passes, then execution. The plan that omits the obstacle diverges from its predicted path. Obstacle includedExecuting the selected actionsgoalSame starting state. Same action budget.Obstacle omittedExecuting the selected actionsprediction gapgoalPrediction and execution separate.
Fig. 04. Two optimizers use the same seed and budget. Green traces show predicted paths; dark traces show execution. The right model omits the obstacle, but the environment still enforces collisions.

Planning once versus replanning

The animation executes a complete chosen sequence so the prediction/execution discrepancy stays visible. In model predictive control, an agent usually executes the first action or a short prefix, obtains another observation, and solves the planning problem again. Replanning can reduce the period for which it relies on an uncorrected prediction.

That does not make MPC immune to a missing obstacle. If every new plan uses a model that excludes an important constraint, the planner can repeat the same mistake. A mechanism for updating the model, detecting failure, or applying a separate constraint is still needed.

The animation uses known simple dynamics and a hand-written objective. It is a visualization of trajectory optimization, not the neural latent planner from TD-MPC2. It makes visible the relationship between the action distribution, objective, selected prediction, and execution.

Three different routes from model to action

Search at decision time. MuZero combines learned dynamics with reward, policy, and value predictions to support tree search. Its model need not reconstruct every observation. MuZero paper.

Optimize a trajectory. TD-MPC and TD-MPC2 use task-oriented latent dynamics and value estimates for local trajectory optimization. Read these after a simple MPC implementation so you can distinguish the learned representation from the planning procedure. TD-MPC, TD-MPC2.

Train behavior in imagination. A Dreamer-style policy learns from predicted trajectories and later chooses actions through the learned actor. This is different from explicitly sampling candidate trajectories at every decision.

Distinguish improving a prediction model, improving a policy, and increasing a planner’s search budget. They can change behavior for different reasons.

06 / The architecture atlas

Why the architectures diverge

The next conceptual jump is realizing that there is no single necessary prediction target. A model can generate observations, predict tokens, preserve task-oriented features, or estimate reward and value. Which target is useful depends on how the model will be used.

The figures below trace what each architecture represents and what its predictions are used for. Start with the complete computation, or follow the animated trace through state inference, training, and decision-making. Each figure names the version it describes. Colors distinguish observations and memory, predicted quantities, actions, and learning objectives; line styles distinguish computation, recurrence, supervision, and parameter updates.

PlaNet → Dreamer

PlaNet and Dreamer combine recurrent memory with a stochastic latent state. The representation has to carry enough of the observation history to predict transitions and task signals when new images are unavailable.

Observation / memoryLatents / predictionsActions / behaviorLearning objective
Forward computationRecurrenceSupervisionParameter update

A state has two parts: memory and uncertainty.

DreamerV3 · one observed step. The image corrects a prediction made from previous experience.

A state has two parts: memory and uncertainty.DreamerV3 · one observed step. The image corrects a prediction made from previous experience. The recurrent transition combines the previous memory, stochastic state, and action. Its new memory predicts a prior over zₜ. The encoder adds the current observation. The posterior uses that evidence and memory; its sampled zₜ joins hₜ to form the model state. Prediction heads learn observation, reward, and continuation targets. Opposite stop-gradient directions balance prior and posterior learning. DreamerV3. Categorical grids are schematic; sg denotes stop-gradient. Free-nats clipping, loss weights, layer widths, and normalization are omitted.Previous statehₜ₋₁, zₜ₋₁, aₜ₋₁fθRecurrenttransitionhₜ = fθ(hₜ₋₁,zₜ₋₁,aₜ₋₁)DeterministicmemoryhₜPriorpθ(zₜ | hₜ)Observationoₜ from replayVisual encodereₜ = enc(oₜ)Posteriorqθ(zₜ | hₜ,eₜ)Model statesₜ = [hₜ,zₜ], zₜ ∼ qθObservationdecoderreconstruct replay oₜReward headlikelihood of replay rₜĉContinue headlikelihood of 1 −terminalKLBalanced KL lossesprior ↔ posterior
A state has two parts: memory and uncertainty.DreamerV3 · one observed step. The image corrects a prediction made from previous experience. The recurrent transition combines the previous memory, stochastic state, and action. Its new memory predicts a prior over zₜ. The encoder adds the current observation. The posterior uses that evidence and memory; its sampled zₜ joins hₜ to form the model state. Prediction heads learn observation, reward, and continuation targets. Opposite stop-gradient directions balance prior and posterior learning. DreamerV3. Categorical grids are schematic; sg denotes stop-gradient. Free-nats clipping, loss weights, layer widths, and normalization are omitted.Previous statehₜ₋₁, zₜ₋₁, aₜ₋₁fθRecurrenttransitionhₜ = fθ(hₜ₋₁,zₜ₋₁,aₜ₋₁)DeterministicmemoryhₜPriorpθ(zₜ | hₜ)Observationoₜ from replayVisual encodereₜ = enc(oₜ)Posteriorqθ(zₜ | hₜ,eₜ)Model statesₜ = [hₜ,zₜ], zₜ ∼ qθObservationdecoderreconstruct replay oₜReward headlikelihood of replay rₜĉContinue headlikelihood of 1 −terminalKLBalanced KL lossesprior ↔ posterior

Ldyn = KL[sg(q) ‖ p] Lrep = KL[q ‖ sg(p)]

DreamerV3. Categorical grids are schematic; sg denotes stop-gradient. Free-nats clipping, loss weights, layer widths, and normalization are omitted.

The actor learns from futures the model invents.

DreamerV3 · imagined trajectories. The same dynamics parameters are reused at each step.

The actor learns from futures the model invents.DreamerV3 · imagined trajectories. The same dynamics parameters are reused at each step. The actor samples an action from the current latent state. The RSSM generates the next state without a new image; the loop repeats. Reward and continuation predictions combine with critic values to produce bootstrapped λ-return targets. The actor is trained using advantage-weighted log probabilities and entropy. The critic fits return targets with slow-value regularization. The diagram follows the DreamerV3 implementation. Imagined features and the actor’s value baseline are detached. Orange inputs carry log-probabilities and entropy; purple inputs carry critic predictions. PlaNet instead searches action sequences online with CEM; it does not use this Dreamer actor-training path.Start stateposterior from replayActor πφsample action aₜfθ, pθRSSM prior stephₜ₊₁, zₜ₊₁ ∼ pθNext imaginedstatesₜ₊₁ → repeat to HPredicted signalsreward, continuationVψCritic Vψbootstrap future valueΣλ-return targetsaccumulate, bootstrapImprove actoradvantage + entropyLVFit criticreturn targets + slow V
The actor learns from futures the model invents.DreamerV3 · imagined trajectories. The same dynamics parameters are reused at each step. The actor samples an action from the current latent state. The RSSM generates the next state without a new image; the loop repeats. Reward and continuation predictions combine with critic values to produce bootstrapped λ-return targets. The actor is trained using advantage-weighted log probabilities and entropy. The critic fits return targets with slow-value regularization. The diagram follows the DreamerV3 implementation. Imagined features and the actor’s value baseline are detached. Orange inputs carry log-probabilities and entropy; purple inputs carry critic predictions. PlaNet instead searches action sequences online with CEM; it does not use this Dreamer actor-training path.Start stateposterior from replayActor πφsample action aₜfθ, pθRSSM prior stephₜ₊₁, zₜ₊₁ ∼ pθNext imaginedstatesₜ₊₁ → repeat to HPredicted signalsreward, continuationVψCritic Vψbootstrap future valueΣλ-return targetsaccumulate, bootstrapImprove actoradvantage + entropyLVFit criticreturn targets + slow V

Deployment: observation → posterior state → actor → environment

The diagram follows the DreamerV3 implementation. Imagined features and the actor’s value baseline are detached. Orange inputs carry log-probabilities and entropy; purple inputs carry critic predictions. PlaNet instead searches action sequences online with CEM; it does not use this Dreamer actor-training path.

PlaNet searches at the moment an action is needed.

PlaNet · the behavior branch diverges here: online planning uses latent rollouts and predicted rewards.

PlaNet searches at the moment an action is needed.PlaNet · the behavior branch diverges here: online planning uses latent rollouts and predicted rewards. Infer a latent state from the current observation history. Initialize a distribution over candidate action sequences. Roll each candidate through the RSSM prior and use the reward model to score its predicted trajectory. CEM refits the action distribution to elite sequences and repeats. Execute only the first action, observe, update the posterior, and plan again. Original PlaNet uses Gaussian stochastic latents. This contrasts with the categorical DreamerV3 state shown above. The planner is CEM in a receding-horizon loop; it is not Dreamer’s actor–critic objective.Inferred statememory + Gaussian latentSample sequencescandidate actions aₜ:ₜ₊HRSSM priorrolloutsfuture states underactionsPredicted rewardsscore each trajectoryCEM refinementfit distribution toelitesExecute firstactiondiscard the rest; observeNew observationupdate posterior state
PlaNet searches at the moment an action is needed.PlaNet · the behavior branch diverges here: online planning uses latent rollouts and predicted rewards. Infer a latent state from the current observation history. Initialize a distribution over candidate action sequences. Roll each candidate through the RSSM prior and use the reward model to score its predicted trajectory. CEM refits the action distribution to elite sequences and repeats. Execute only the first action, observe, update the posterior, and plan again. Original PlaNet uses Gaussian stochastic latents. This contrasts with the categorical DreamerV3 state shown above. The planner is CEM in a receding-horizon loop; it is not Dreamer’s actor–critic objective.Inferred statememory + GaussianlatentSample sequencescandidate actionsaₜ:ₜ₊HRSSM priorrolloutsfuture states underactionsPredicted rewardsscore each trajectoryCEM refinementfit distribution toelitesExecute firstactiondiscard the rest;observeNew observationupdate posterior state

PlaNet: optimize actions online. Dreamer: train an actor in imagination.

Original PlaNet uses Gaussian stochastic latents. This contrasts with the categorical DreamerV3 state shown above. The planner is CEM in a receding-horizon loop; it is not Dreamer’s actor–critic objective.

Original explanatory diagrams. Scenes, tensor grids, and candidate curves are schematic. Open complete SVG ↗

Diagram sources: PlaNet paper · planningDreamer paper · behavior learningDreamerV3 code · Agent.loss / policyDreamerV3 RSSM · observe / imagine / loss

The main change is how imagined experience produces behavior: online search in PlaNet, actor-critic learning in Dreamer.

Their use of imagination differs. PlaNet searches over action sequences in latent space. Dreamer uses imagined trajectories to train an actor and critic. A good reconstruction is useful training evidence, but the eventual test is behavior in the environment.

PlaNet ↗Dreamer ↗DreamerV2 ↗DreamerV3 ↗Dreamer 4 ↗

MuZero / TD-MPC2

MuZero learns a representation whose transitions support reward, policy, and value prediction. Tree search uses those quantities to compare possible actions. Reconstructing the full observation is unnecessary for that planning procedure.

Observation / memoryLatents / predictionsActions / behaviorLearning objective
Forward computationRecurrenceSupervisionParameter update

MuZero builds a search tree inside a learned state.

MuZero · representation h, dynamics g, prediction f. Rewards belong to transitions; policies and values belong to states.

MuZero builds a search tree inside a learned state.MuZero · representation h, dynamics g, prediction f. Rewards belong to transitions; policies and values belong to states. Representation h maps the observation history to a root state. This state is trained for search-relevant predictions. Each hypothetical action goes through g, producing the next hidden state and reward. Function f supplies policy priors and a value at that state. Search repeatedly selects, expands, and backs up returns. Root visits determine the search policy used to choose the real action. Branches show different hypothetical actions with shared g and f parameters. Joint training unrolls real replay actions and fits policy, value, and reward targets. No observation reconstruction objective is required.Observationhistoryo₁ … oₜRepresentation hhistory → root s⁰Root state s⁰start of the searchvRoot prediction fpolicy priors + valuegDynamics g ·action a(s⁰,a) → (s¹,r¹)gDynamics g ·action b(s⁰,b) → (s¹,r¹)vPrediction f(s¹)policy prior p, value vvPrediction f(s¹)policy prior p, value vExpand + back updeeper g steps; returnsRoot visit countschoose an action
MuZero builds a search tree inside a learned state.MuZero · representation h, dynamics g, prediction f. Rewards belong to transitions; policies and values belong to states. Representation h maps the observation history to a root state. This state is trained for search-relevant predictions. Each hypothetical action goes through g, producing the next hidden state and reward. Function f supplies policy priors and a value at that state. Search repeatedly selects, expands, and backs up returns. Root visits determine the search policy used to choose the real action. Branches show different hypothetical actions with shared g and f parameters. Joint training unrolls real replay actions and fits policy, value, and reward targets. No observation reconstruction objective is required.Observationhistoryo₁ … oₜRepresentation hhistory → root s⁰Root state s⁰start of the searchvRoot prediction fpolicy priors + valuegDynamics g ·action a(s⁰,a) → (s¹,r¹)gDynamics g ·action b(s⁰,b) → (s¹,r¹)vPrediction f(s¹)policy prior p, value vvPrediction f(s¹)policy prior p, value vExpand + back updeeper g steps; returnsRoot visit countschoose an action

Training targets: search policy π, return z, observed reward u

Branches show different hypothetical actions with shared g and f parameters. Joint training unrolls real replay actions and fits policy, value, and reward targets. No observation reconstruction objective is required.

TD-MPC2 optimizes a short sequence of actions.

TD-MPC2 · continuous control. Candidate trajectories are scored using rewards and a terminal value estimate.

TD-MPC2 optimizes a short sequence of actions.TD-MPC2 · continuous control. Candidate trajectories are scored using rewards and a terminal value estimate. Encode the current observation. The policy contributes action-sequence proposals alongside sampled candidates. The latent dynamics model rolls each sequence forward. Predicted rewards and an ensemble-based terminal Q estimate determine its score. Weighted elites update the sampling distribution. Refine again, then execute only the first action and replan after observing the environment. Curves illustrate candidates, not benchmark results. Training combines latent consistency, reward regression, and bootstrapped Q objectives. Task embeddings and implementation-specific value aggregation are omitted.Observationcurrent state oₜEncoderzₜ = enc(oₜ)Policy proposalscandidate action pathsSample + roll outz′ = d(z,a)Reward modelr(z,a) along trajectoryQQ ensembleterminal Q(zH,π(zH))ΣTrajectory scorerewards + terminal QRefinedistributionweighted elite sequencesExecute firstactionobserve again; replan
TD-MPC2 optimizes a short sequence of actions.TD-MPC2 · continuous control. Candidate trajectories are scored using rewards and a terminal value estimate. Encode the current observation. The policy contributes action-sequence proposals alongside sampled candidates. The latent dynamics model rolls each sequence forward. Predicted rewards and an ensemble-based terminal Q estimate determine its score. Weighted elites update the sampling distribution. Refine again, then execute only the first action and replan after observing the environment. Curves illustrate candidates, not benchmark results. Training combines latent consistency, reward regression, and bootstrapped Q objectives. Task embeddings and implementation-specific value aggregation are omitted.Observationcurrent state oₜEncoderzₜ = enc(oₜ)Policy proposalscandidate action pathsSample + roll outz′ = d(z,a)Reward modelr(z,a) along trajectoryQQ ensembleterminal Q(zH,π(zH))ΣTrajectory scorerewards + terminal QRefinedistributionweighted elitesequencesExecute firstactionobserve again; replan

J = Σₖ γᵏ r(zk,ak) + γᴴ Q(zH,π(zH))

Curves illustrate candidates, not benchmark results. Training combines latent consistency, reward regression, and bootstrapped Q objectives. Task embeddings and implementation-specific value aggregation are omitted.

Original explanatory diagrams. Scenes, tensor grids, and candidate curves are schematic. Open complete SVG ↗

Diagram sources: MuZero paper · Figure 1TD-MPC2 model codeTD-MPC2 planner · _estimate_value / _plan

The observation decoder drops out of the training objective. Reward and value predictions shape a state useful for search.

TD-MPC2 also learns a task-oriented latent model, but uses local trajectory optimization and value estimates for continuous control. Grouping it with MuZero highlights their task-oriented objectives; their search procedures and training algorithms differ.

MuZero ↗TD-MPC ↗TD-MPC2 ↗

IRIS

IRIS turns images into discrete visual tokens and models their evolution with an autoregressive Transformer. Actions enter the sequence, so the predicted continuation depends on the agent’s choices.

Observation / memoryLatents / predictionsActions / behaviorLearning objective
Forward computationRecurrenceSupervisionParameter update

A frame becomes a vocabulary; dynamics become a sequence.

Original IRIS · a visual tokenizer and a causal Transformer have separate training objectives.

A frame becomes a vocabulary; dynamics become a sequence.Original IRIS · a visual tokenizer and a causal Transformer have separate training objectives. The visual encoder and codebook turn each image into discrete IDs. Actions occupy their own positions in the temporal sequence. A decoder trains the visual vocabulary through reconstruction. The causal Transformer receives token IDs; its objective does not update the tokenizer. Separate heads predict observation tokens, reward, and episode end. Cross-entropy trains these discrete predictions. Grid size is illustrative; K depends on the tokenizer. Action embeddings and observation embeddings are separate. The causal mask prevents access to future sequence positions.Observation64 × 64 RGB frameVisual encodercontinuous feature mapVectorquantizationnearest codebook entriesInterleavedsequencezₜ¹ … zₜᴷ, aₜ, zₜ₊₁¹ …Tokenizer decoderreconstruct the inputLtokTokenizerobjectivereconstruction + VQlossescausal attnfeed-forwardresidual + normCausal Transformerpredict the next tokenObservation headnext visual-token logitsReward headreward classificationendEnd headepisode-end probability
A frame becomes a vocabulary; dynamics become a sequence.Original IRIS · a visual tokenizer and a causal Transformer have separate training objectives. The visual encoder and codebook turn each image into discrete IDs. Actions occupy their own positions in the temporal sequence. A decoder trains the visual vocabulary through reconstruction. The causal Transformer receives token IDs; its objective does not update the tokenizer. Separate heads predict observation tokens, reward, and episode end. Cross-entropy trains these discrete predictions. Grid size is illustrative; K depends on the tokenizer. Action embeddings and observation embeddings are separate. The causal mask prevents access to future sequence positions.Observation64 × 64 RGB frameVisual encodercontinuous feature mapVectorquantizationnearest codebookentriesInterleavedsequencezₜ¹ … zₜᴷ, aₜ, zₜ₊₁¹ …Tokenizer decoderreconstruct the inputLtokTokenizerobjectivereconstruction + VQlossescausal attnfeed-forwardresidual + normCausal Transformerpredict the next tokenObservation headnext visual-tokenlogitsReward headreward classificationendEnd headepisode-end probability

… [zₜ¹, …, zₜᴷ, aₜ] [zₜ₊₁¹, …, zₜ₊₁ᴷ, aₜ₊₁] …

Grid size is illustrative; K depends on the tokenizer. Action embeddings and observation embeddings are separate. The causal mask prevents access to future sequence positions.

The policy sees an image decoded from imagination.

Original IRIS · the actor–critic interacts with the world model through an image interface.

The policy sees an image decoded from imagination.Original IRIS · the actor–critic interacts with the world model through an image interface. Predicted visual tokens are decoded back to pixels before the actor consumes them. The actor’s convolutional network processes the imagined image. An LSTM retains temporal context. The actor supplies the next action token to the world model. Reward and end predictions supply the imagined learning signals. The action enters the next Transformer step. The imagined environment is evaluated without actor gradients through the world model. Layer counts follow the original IRIS implementation.Sample visualtokensworld-model continuationTokenizer decodertokens → RGB frameImagined frame64 × 64 × 3Actor visualnetwork4 convolution/pool layerscausal attnfeed-forwardresidual + normWorld-model stepsample z; predict r / endLSTM memory512-dimensional stateActor headnext environment actionVValue headtrain actor–critic
The policy sees an image decoded from imagination.Original IRIS · the actor–critic interacts with the world model through an image interface. Predicted visual tokens are decoded back to pixels before the actor consumes them. The actor’s convolutional network processes the imagined image. An LSTM retains temporal context. The actor supplies the next action token to the world model. Reward and end predictions supply the imagined learning signals. The action enters the next Transformer step. The imagined environment is evaluated without actor gradients through the world model. Layer counts follow the original IRIS implementation.Sample visualtokensworld-modelcontinuationTokenizer decodertokens → RGB frameImagined frame64 × 64 × 3Actor visualnetwork4 convolution/poollayerscausal attnfeed-forwardresidual + normWorld-model stepsample z; predict r /endLSTM memory512-dimensional stateActor headnext environment actionVValue headtrain actor–critic

The action enters the next Transformer step. The imagined environment is evaluated without actor gradients through the world model. Layer counts follow the original IRIS implementation.

Original explanatory diagrams. Scenes, tensor grids, and candidate curves are schematic. Open complete SVG ↗

Diagram sources: IRIS paper · model overviewPrediction heads · world_model.pyDecode path · world_model_env.pyIRIS actor–critic codeIRIS tokenizer objectives

The temporal state becomes a sequence of discrete visual and action tokens. Imagined frames are decoded before the actor-critic sees them.

The tokenizer determines which visual distinctions the temporal model can retain. The resulting imagined experience trains an actor-critic. Token prediction error and real-environment return measure different parts of that system.

IRIS paper ↗Official code ↗

JEPA / V-JEPA

JEPA-style models predict learned representations of masked or future content. Pixel reconstruction is not the training target. This shifts attention toward which information the representation preserves and which downstream tasks can use it.

Observation / memoryLatents / predictionsActions / behaviorLearning objective
Forward computationRecurrenceSupervisionParameter update

Two branches meet in a feature-prediction loss.

V-JEPA 2 · action-free pretraining. A moving-average teacher supplies the target representation.

Two branches meet in a feature-prediction loss.V-JEPA 2 · action-free pretraining. A moving-average teacher supplies the target representation. The student encodes the visible context. A separate encoder sees the full video and follows the student by exponential moving average. Mask-location embeddings tell the predictor which missing regions to estimate from context features. The loss compares predicted and target representations. Gradients train the student and predictor; the target branch uses stop-gradient. Dashed connection: parameter update by EMA, not a video-feature connection. Pretraining is action-free. It predicts representations rather than reconstructing pixels.Visible videopatchesmasked contextContext encodertrainable ViTself-attentionfeed-forwardresidual + normPredictorfeatures + mask positionsPredicted featurestarget locationsFull videoincludes target regionsTarget encoderEMA of context encoderTarget featuresstop gradientLFeature lossnormalized target patches
Two branches meet in a feature-prediction loss.V-JEPA 2 · action-free pretraining. A moving-average teacher supplies the target representation. The student encodes the visible context. A separate encoder sees the full video and follows the student by exponential moving average. Mask-location embeddings tell the predictor which missing regions to estimate from context features. The loss compares predicted and target representations. Gradients train the student and predictor; the target branch uses stop-gradient. Dashed connection: parameter update by EMA, not a video-feature connection. Pretraining is action-free. It predicts representations rather than reconstructing pixels.Visible videopatchesmasked contextContext encodertrainable ViTself-attentionfeed-forwardresidual + normPredictorfeatures + maskpositionsPredicted featurestarget locationsFull videoincludes target regionsTarget encoderEMA of context encoderTarget featuresstop gradientLFeature lossnormalized targetpatches

θtarget ← τ θtarget + (1 − τ) θcontext

Dashed connection: parameter update by EMA, not a video-feature connection. Pretraining is action-free. It predicts representations rather than reconstructing pixels.

Actions enter during a separate adaptation stage.

V-JEPA 2-AC · inference and planning after adaptation. Frozen image features feed an action-conditioned predictor.

Actions enter during a separate adaptation stage.V-JEPA 2-AC · inference and planning after adaptation. Frozen image features feed an action-conditioned predictor. Freeze the pretrained visual encoder. Encode the current context and the goal, while candidate actions and robot state supply control inputs. The adapted predictor rolls features forward. A distance to the encoded goal scores the hypothetical outcome. CEM refines action candidates inside model predictive control. Execute the first action, observe again, and repeat. The action-conditioned predictor is trained on robot trajectories with teacher-forced and two-step rollout L1 losses. The paper’s robot configuration uses 16×16×1408 encoder features and a 24-layer, 16-head, width-1024 predictor with 7D action/state inputs. It is not the action-free pretraining predictor.Current imagerobot observation xₖFrozen encodercurrent state featuresActions + robotstatecandidate sequenceblock-causal attnfeed-forwardresidual + normAC predictorroll future featuresGoal imagedesired visual outcomeSame frozenencodergoal featuresL1Goal-featuredistancepredicted state vs goalCEM / MPCrefine; execute firstaction
Actions enter during a separate adaptation stage.V-JEPA 2-AC · inference and planning after adaptation. Frozen image features feed an action-conditioned predictor. Freeze the pretrained visual encoder. Encode the current context and the goal, while candidate actions and robot state supply control inputs. The adapted predictor rolls features forward. A distance to the encoded goal scores the hypothetical outcome. CEM refines action candidates inside model predictive control. Execute the first action, observe again, and repeat. The action-conditioned predictor is trained on robot trajectories with teacher-forced and two-step rollout L1 losses. The paper’s robot configuration uses 16×16×1408 encoder features and a 24-layer, 16-head, width-1024 predictor with 7D action/state inputs. It is not the action-free pretraining predictor.Current imagerobot observation xₖFrozen encodercurrent state featuresActions + robotstatecandidate sequenceblock-causal attnfeed-forwardresidual + normAC predictorroll future featuresGoal imagedesired visual outcomeSame frozenencodergoal featuresL1Goal-featuredistancepredicted state vs goalCEM / MPCrefine; execute firstaction

The action-conditioned predictor is trained on robot trajectories with teacher-forced and two-step rollout L1 losses. The paper’s robot configuration uses 16×16×1408 encoder features and a 24-layer, 16-head, width-1024 predictor with 7D action/state inputs. It is not the action-free pretraining predictor.

Original explanatory diagrams. Scenes, tensor grids, and candidate curves are schematic. Open complete SVG ↗

Diagram sources: V-JEPA 2 paper · Sections 2–4Pretraining · EMA and target branchAction-conditioned training code

Prediction is compared in feature space. Robot actions enter in the separate V-JEPA 2-AC stage, not in action-free pretraining.

An action interface depends on the variant. V-JEPA 2-AC explicitly adds action-conditioned prediction for robot planning; a generic representation checkpoint does not imply the same capability.

I-JEPA ↗V-JEPA 2 ↗V-JEPA 2.1 ↗

Genie

Genie generates visual continuations that respond to actions. The original system learns latent actions from video; later releases develop interactive generated environments. A compressed visual history supplies the context for those continuations.

Observation / memoryLatents / predictionsActions / behaviorLearning objective
Forward computationRecurrenceSupervisionParameter update

Genie learns what an action means from video.

Original Genie (2024) · a video tokenizer and a separate latent action model establish two discrete vocabularies.

Genie learns what an action means from video.Original Genie (2024) · a video tokenizer and a separate latent action model establish two discrete vocabularies. Train the spatiotemporal video tokenizer to reconstruct clips through a discrete visual codebook. The latent action encoder sees the transition, including the next frame. Its decoder reconstructs that next frame from the past and the inferred action code. Dynamics learns masked visual-token prediction using action embeddings. The action embedding is detached: dynamics loss does not train the action model. Tokenizer training precedes joint latent-action/dynamics training; the original action vocabulary has eight codes. Action embeddings are added to dynamics representations; they are not interleaved action tokens as in IRIS.Training clippast frames + next frameST video tokenizercompress visual framesVisual codebookframe tokens zVideo decodertokenizer reconstructionLatent actionencoderinfer transition fromframesAction codebookdiscrete latent action ãLatent actiondecoderpast frames + ã → nextframeLLAMAction-model lossreconstruction + VQspatial attncausal temporalfeed-forwardST dynamics modelvisual tokens + sg(ã)CEDynamics lossmasked-tokencross-entropy
Genie learns what an action means from video.Original Genie (2024) · a video tokenizer and a separate latent action model establish two discrete vocabularies. Train the spatiotemporal video tokenizer to reconstruct clips through a discrete visual codebook. The latent action encoder sees the transition, including the next frame. Its decoder reconstructs that next frame from the past and the inferred action code. Dynamics learns masked visual-token prediction using action embeddings. The action embedding is detached: dynamics loss does not train the action model. Tokenizer training precedes joint latent-action/dynamics training; the original action vocabulary has eight codes. Action embeddings are added to dynamics representations; they are not interleaved action tokens as in IRIS.Training clippast frames + nextframeST video tokenizercompress visual framesVisual codebookframe tokens zVideo decodertokenizerreconstructionLatent actionencoderinfer transition fromframesAction codebookdiscrete latent actionãLatent actiondecoderpast frames + ã → nextframeLLAMAction-model lossreconstruction + VQspatial attncausal temporalfeed-forwardST dynamics modelvisual tokens + sg(ã)CEDynamics lossmasked-tokencross-entropy

Tokenizer training precedes joint latent-action/dynamics training; the original action vocabulary has eight codes. Action embeddings are added to dynamics representations; they are not interleaved action tokens as in IRIS.

At inference, the user supplies the action code.

Original Genie · generate one frame’s tokens, decode it, then append it to history.

At inference, the user supplies the action code.Original Genie · generate one frame’s tokens, decode it, then append it to history. Encode the available visual history and choose a learned action code. No future frame is available to infer the action at inference. The dynamics Transformer uses the selected action embedding and past visual tokens to predict masked sites in the next frame. MaskGIT refines the frame’s token grid. Decode the completed grid and append its tokens to history before taking the next action. The latent action encoder/decoder are discarded for inference; the codebook remains. Generation is autoregressive across frames, with iterative token refinement within each frame.Visual tokenhistoryz₁ … zₜSelected actioncodecodebook entry ãₜspatial attncausal temporalfeed-forwardST Transformercondition on history + ãₜMasked next frameiteratively fill tokensitesCompleted frametokensMaskGIT refinementVideo decodervisual tokens → pixelsGenerated framedisplay + extend history
At inference, the user supplies the action code.Original Genie · generate one frame’s tokens, decode it, then append it to history. Encode the available visual history and choose a learned action code. No future frame is available to infer the action at inference. The dynamics Transformer uses the selected action embedding and past visual tokens to predict masked sites in the next frame. MaskGIT refines the frame’s token grid. Decode the completed grid and append its tokens to history before taking the next action. The latent action encoder/decoder are discarded for inference; the codebook remains. Generation is autoregressive across frames, with iterative token refinement within each frame.Visual tokenhistoryz₁ … zₜSelected actioncodecodebook entry ãₜspatial attncausal temporalfeed-forwardST Transformercondition on history +ãₜMasked next frameiteratively fill tokensitesCompleted frametokensMaskGIT refinementVideo decodervisual tokens → pixelsGenerated framedisplay + extendhistory

The latent action encoder/decoder are discarded for inference; the codebook remains. Generation is autoregressive across frames, with iterative token refinement within each frame.

Original explanatory diagrams. Scenes, tensor grids, and candidate curves are schematic. Open complete SVG ↗

Diagram sources: Genie paper · components and inference

The action labels can be learned from video. At inference, a user-selected code replaces the action inferred from the next frame.

The visual output exposes both the appeal and the difficulty of this approach. A scene can look convincing while forgetting an object or changing its mechanics. Persistence, action sensitivity, and downstream usefulness require separate evaluation.

Genie paper ↗Genie 2 ↗Genie 3 ↗

DayDreamer / Cosmos

Physical systems make the action interface specific: actuator commands, sensory delays, contacts, and the body’s own geometry affect the transition. DayDreamer studies world-model learning on real robots.

Observation / memoryLatents / predictionsActions / behaviorLearning objective
Forward computationRecurrenceSupervisionParameter update

DayDreamer closes the loop through a real robot.

DayDreamer · separate the physical interaction loop from model and behavior learning.

DayDreamer closes the loop through a real robot.DayDreamer · separate the physical interaction loop from model and behavior learning. Robot observations update the latent state. The actor selects a command, which changes the physical environment and produces the next observation. Recorded interaction trains the world model: recurrent dynamics and observation/reward/continuation predictions. Imagined trajectories train actor and critic. Updated actor parameters return to the robot interaction loop. Dashed connections denote the actuator loop and an actor-parameter update. This is the DayDreamer learning architecture; the body-specific sensor and actuator interfaces vary by experiment.Physical robotsensors and actuatorsState inferenceencode + RSSM posteriorActorsample actuator commandReplay experienceobservations, actions,rewardsWorld-modellearningRSSM + prediction headsLatent imaginationrollouts under the actorBehavior learningactor + critic objectives
DayDreamer closes the loop through a real robot.DayDreamer · separate the physical interaction loop from model and behavior learning. Robot observations update the latent state. The actor selects a command, which changes the physical environment and produces the next observation. Recorded interaction trains the world model: recurrent dynamics and observation/reward/continuation predictions. Imagined trajectories train actor and critic. Updated actor parameters return to the robot interaction loop. Dashed connections denote the actuator loop and an actor-parameter update. This is the DayDreamer learning architecture; the body-specific sensor and actuator interfaces vary by experiment.Physical robotsensors and actuatorsState inferenceencode + RSSM posteriorActorsample actuator commandReplay experienceobservations, actions,rewardsWorld-modellearningRSSM + prediction headsLatent imaginationrollouts under theactorBehavior learningactor + criticobjectives

Dashed connections denote the actuator loop and an actor-parameter update. This is the DayDreamer learning architecture; the body-specific sensor and actuator interfaces vary by experiment.

Cosmos-Predict2.5 generates video by integrating a flow.

Cosmos-Predict2.5/base · text and visual context condition a video diffusion Transformer trained with flow matching.

Cosmos-Predict2.5 generates video by integrating a flow.Cosmos-Predict2.5/base · text and visual context condition a video diffusion Transformer trained with flow matching. Clean reference latents replace the initial noisy frames with a condition mask. Text supplies cross-attention; time conditions adaptive normalization. The DiT predicts a velocity field on noisy video latents. Repeated integration steps move toward a generated latent video. The video tokenizer’s decoder turns the final latent into frames. A downstream controller or evaluator is a separate system. This is the base Predict2.5 generation path, not Cosmos 3 or a robot actuator policy. Wan2.1 VAE compression is 4×8×8 in time/height/width; latent patches are 1×2×2. Fields and frames are illustrative. Text encoder internals and solver choices are omitted.Visual contextimage / videoconditioningVideo tokenizercontinuous video latentsText embeddingCosmos-Reason1 featuresReference + noiseclean context; noisyfuture3D self-attntext cross-attnMLPVideo DiTvelocity;time-conditionedFlow integrationupdate latent; repeatTokenizer decoderlatent → video pixelsGenerated videoconditioned continuation
Cosmos-Predict2.5 generates video by integrating a flow.Cosmos-Predict2.5/base · text and visual context condition a video diffusion Transformer trained with flow matching. Clean reference latents replace the initial noisy frames with a condition mask. Text supplies cross-attention; time conditions adaptive normalization. The DiT predicts a velocity field on noisy video latents. Repeated integration steps move toward a generated latent video. The video tokenizer’s decoder turns the final latent into frames. A downstream controller or evaluator is a separate system. This is the base Predict2.5 generation path, not Cosmos 3 or a robot actuator policy. Wan2.1 VAE compression is 4×8×8 in time/height/width; latent patches are 1×2×2. Fields and frames are illustrative. Text encoder internals and solver choices are omitted.Visual contextimage / videoconditioningVideo tokenizercontinuous videolatentsText embeddingCosmos-Reason1 featuresReference + noiseclean context; noisyfuture3D self-attntext cross-attnMLPVideo DiTvelocity;time-conditionedFlow integrationupdate latent; repeatTokenizer decoderlatent → video pixelsGenerated videoconditionedcontinuation

Training: regress a target flow velocity at sampled interpolation times

This is the base Predict2.5 generation path, not Cosmos 3 or a robot actuator policy. Wan2.1 VAE compression is 4×8×8 in time/height/width; latent patches are 1×2×2. Fields and frames are illustrative. Text encoder internals and solver choices are omitted.

Original explanatory diagrams. Scenes, tensor grids, and candidate curves are schematic. Open complete SVG ↗

Diagram sources: DayDreamer paperDayDreamer agent codeCosmos-Predict2.5 reportCosmos base inference interfacesPredict2.5 rectified-flow code

Embodiment specifies sensors and actuators. It does not imply a single architecture: learning a robot policy and generating a video serve different roles.

Cosmos covers several model-specific workflows around world generation, transformation, and reasoning. Comparing embodied systems requires stating which sensors and actions are available, how much real interaction they use, and where the resulting behavior is evaluated.

DayDreamer ↗RoboDreamer ↗Cosmos documentation ↗Cosmos 3 report ↗

A Transformer does not settle the representation question

IRIS treats dynamics as sequence modeling over discrete visual representations with an autoregressive Transformer. This brings tokenization and context into the world-model problem. The temporal model can only work with the information its visual tokens preserve.

A low token-prediction loss does not by itself answer whether the model supports good decisions. Inspect reconstruction errors, recursive rollouts, the action interface, and behavior in the actual task.

Predicting features is a different commitment

JEPA-style objectives predict representations rather than directly reconstructing every pixel. Depending on the variant, the target can concern masked regions or temporal structure. V-JEPA 2 connects representation learning with prediction and, in its action-conditioned 2-AC variant, robot planning. V-JEPA 2.1 develops dense visual representations.

The practical question is what those features preserve and how a downstream system uses them. A representation checkpoint is not automatically an action-conditioned simulator or a complete controller.

Generating a world introduces another set of tests

Genie’s original paper learns controllable environments from video using a latent action model. Genie 2 and Genie 3 expand the interactive generation direction. OpenAI’s original Sora technical report studies large-scale video generation and documents both simulator-like capabilities and limitations. That report is a historical architectural reference, not a description of every subsequent Sora release.

The key questions are whether the model responds to changed actions, maintains relevant state when it leaves the view, and supports a measured downstream task. Attractive video and reliable closed-loop dynamics require different evidence.

Embodiment makes the interface concrete

DayDreamer brings world-model behavior learning to physical robots. RoboDreamer studies compositional world models for robot imagination. The Cosmos ecosystem spans model-specific generation, transformation, and reasoning workflows; its Cosmos 3 research page is an additional current reference.

For any embodied system, inspect the sensory inputs, action representation, embodiment, data coverage, and evaluation environment. Game actions, latent video actions, and robot actuator commands are different interfaces. A successful result under one interface does not establish transfer to another.

Research snapshot checked October 10, 2026. The atlas is a selection of approaches. It does not rank incompatible benchmarks or assume that the newest name supersedes every older conceptual tool.

07 / Uncertainty and evaluation

Where predictions break down

Every imagined transition depends on the predictive state it receives. In a recursive rollout, an error can change the next input, which changes the next prediction. The result may be a trajectory outside the training distribution.

But “model error” does not name one universal phenomenon. Measurement noise can make the state ambiguous. Uncertain parameters can produce disagreement between dynamics models. A misspecified model class can make all models agree on an incorrect result. A policy can steer the system into states that the dataset scarcely covers.

Thirteen spring models below share an initial state but disagree about stiffness. Their mean stiffness is also different from the true system. One group rolls forward without correction. The other receives the exact state every eight steps, or 0.4 seconds. Its predictions repeatedly return to the observed trajectory, although the incorrect parameter assumptions remain.

Figure 5: The same thirteen spring models roll forward with and without exact state observations every eight steps. Blue spread shows their range, not a calibrated confidence interval. Two animated fans of spring trajectories compare uncorrected prediction drift with state correction every 0.4 seconds. The dark line shows the true system. Without fresh observationsThirteen models, slightly different stiffness06 sWith fresh observationsState corrected every 0.4 seconds06 s
Fig. 05. The same thirteen spring models roll forward with and without exact state observations every eight steps. Blue spread shows their range, not a calibrated confidence interval.

Spread is not automatically confidence

The band shows the minimum and maximum of these thirteen selected trajectories. It is not a 95% interval, and the number of members does not make it calibrated. Reducing their parameter spread can make the models more confident-looking while their shared assumption remains wrong.

Epistemic uncertainty concerns missing knowledge about the model or its parameters. Aleatoric uncertainty concerns variability in the observation or transition process. Separating them requires an explicit modeling assumption; a fan of arbitrary deterministic models does not provide a full uncertainty decomposition.

PETS is a useful reference for ensembles and uncertainty propagation. MBPO is a useful reference for balancing additional imagined experience against model error. Read them alongside experiments that vary horizon and dataset coverage.

An evaluation ladder

CapabilityControlled comparisonUseful measurements
Predict observed transitionsHeld-out episodes; compare one-step and recursive predictionState error against horizon; failure slices
Preserve task-relevant structureOcclusions, contacts, and persistent object changesConsistency under the chosen task’s rules
Respond to actionsFix starting context and change the action sequenceCorrect sensitivity to action and constraint changes
Express uncertaintyTest covered and shifted conditionsCalibration, coverage, disagreement, and error together
Improve decisionsMatch real data, compute, and task evaluation budgetsReal return or success, interaction cost, and recovery

Report which capability your result actually tests. A demonstration of interactive response is useful, but it is not the same as demonstrating a better planning policy.

Conditioning and intervention

Action-conditioned prediction, p(s′ | s, a), is not automatically a causally identified intervention model, p(s′ | s, do(a)). If actions in the dataset depend on hidden circumstances, a model can learn correlations that fail when a different policy selects actions.

The practical response is to examine data collection, vary actions under controlled conditions where possible, and describe the assumptions behind a causal claim. A model’s input having an “action” field is not sufficient evidence of intervention validity.

Three research extensions of the same problem

Other agents change the transition. A pedestrian, opponent, or collaborating robot chooses actions too. A predictor may need to represent another agent’s intention or policy, and the behavior it observes can change in response to the learner. Test a changed counterpart policy rather than assuming training-time behavior remains fixed.

Embodiment changes what an action means. The same abstract instruction can correspond to different actuator ranges, kinematics, delays, or contact dynamics. A shared representation can help, but cross-embodiment evaluation must specify how observations and actions are aligned. A result with one robot is evidence about that setup, not automatic evidence about every body.

Driving adds representation choices. Future images, object trajectories, occupancy, and geometry expose different errors and support different downstream decisions. Preserve the coordinate system, action interface, and evaluated horizon when comparing models. An accurate-looking scene is only one part of a driving task’s evidence.

These extensions are ways to turn the same state/prediction/action questions into research protocols. The surveys below provide a broader map of embodiments, representations, and evaluation settings.

The questions that remain open

Long-horizon memory, uncertainty calibration, multi-agent interaction, changing embodiments, physical consistency, compute cost, and failure recovery all complicate a general world model. These problems interact: adding memory can change state estimation; changing the embodiment can change action semantics; reducing latency can constrain prediction detail.

The comprehensive embodied-AI survey organizes approaches by functionality, temporal modeling, and spatial representation. The actionability survey puts behavior improvement at the center. The spring and obstacle examples give their categories concrete counterparts: state error, model mismatch, and the cost of acting on a wrong prediction.

can every model agree and still be wrong?

Yes. An ensemble can share a missing variable, incorrect model class, or dataset bias. Agreement can measure similarity of the chosen models rather than correctness. Evaluate predictions and decisions against held-out or real observations.

08 / Build and investigate

Experiments worth extending

The small systems in this article suggest several implementation projects. Each makes one failure visible: hidden velocity, accumulating rollout error, missing constraints, or agreement between incorrect models. The list below includes an output to inspect and a comparison to run.

A learner interested in robotics can spend more time on estimation and MPC. A learner interested in visual representation can branch toward JEPA or tokenized sequence models. Neither needs to treat all architecture families as one mandatory chronological queue.

Ten implementation ideas, with suggested checks

  1. A tabular world

    GridWorld + transition counts + Dyna-Q

    Produce
    Plot real and model-generated transitions.
    Test
    Compare equal real-interaction budgets; test an unseen transition.

  2. Identify simple physics

    CartPole or a controlled spring + learned dynamics + MPC

    Produce
    One-step and recursive error curves.
    Test
    Split by trajectory and test a different starting-state range.

  3. Make uncertainty explicit

    Probabilistic outputs + an ensemble + rollout sampling

    Produce
    Reliability and ensemble-disagreement plots.
    Test
    Separate observation noise from model disagreement; check calibration.

  4. Predict from pixels

    Encoder + temporal dynamics + decoder

    Produce
    Observed and imagined frames for the same action sequence.
    Test
    Report dynamics errors as well as image similarity.

  5. A small imagination learner

    RSSM + reward + continuation + actor + critic

    Produce
    Annotated training graph and actual environment returns.
    Test
    Disable imagination and match the data budget for the baseline.

  6. Learn only what search needs

    Small board game + representation/dynamics + tree search

    Produce
    A search tree with reward, policy, and value annotations.
    Test
    Compare learned-model search to known-rule search.

  7. Treat dynamics as a language

    Visual tokenizer + action tokens + Transformer

    Produce
    Token reconstructions and long rollout failures.
    Test
    Vary context length and isolate tokenizer error.

  8. Predict features instead

    Context encoder + target encoder + predictive objective

    Produce
    A pixel-versus-feature comparison under a shared task.
    Test
    Measure collapse and useful downstream distinctions.

  9. Control generated futures

    Small action-labeled video environment

    Produce
    Paired continuations from the same context under changed actions.
    Test
    Keep context fixed and quantify the action’s effect.

  10. Ask one research question

    Uncertainty-aware planning with memory and recovery

    Produce
    A reproducible report with baselines, seeds, budgets, and failures.
    Test
    Show whether the proposed mechanism improves real decisions.

Design the research question before scaling the system

“Build a general world model” leaves too many variables changing together. A more tractable question is whether a specified uncertainty mechanism reduces a specified planning failure under a fixed data budget.

That comparison needs a fixed environment, action space, planning horizon, and data budget. Its outcome is the behavior in the environment. A lower predicted cost alone could mean the planner has become better at exploiting the model’s errors.

A useful report contains the hypothesis, implementation, dataset and evaluation splits, matched budgets, seeds, uncertainty in the estimates, and representative failures. Publish the smallest reproducible artifact that lets another reader inspect the mechanism.

A reading sequence

  1. Where does simulated experience enter learning? Dyna, then a small tabular implementation.
  2. How should uncertain dynamics enter a decision? PILCO, PETS, and MBPO.
  3. What is a useful latent predictive state? World Models, PlaNet, and the Dreamer sequence.
  4. Which predictions are sufficient for search? MuZero and TD-MPC2.
  5. What changes when dynamics become a token sequence? IRIS.
  6. What does representation prediction preserve? JEPA and V-JEPA variants.
  7. What does controllable generation actually demonstrate? Genie and model-specific video-generation reports.
  8. What changes in physical systems? DayDreamer, robot-imagination work, and Cosmos’s model-specific interfaces.
  9. What would count as actionability? The two surveys, revisited with evidence from your own implementation.

These papers become easier to compare when the same questions stay in view: what the state contains, where actions enter, what gets predicted, and how that prediction affects behavior. Architecture names alone obscure those differences.

The missing-obstacle example is a small version of a broader problem. Better search cannot recover a constraint the model never represents. A useful next experiment would change what the model knows, hold the planning budget fixed, and measure whether the executed behavior improves.

References

Further reading.

Books and courses covering the mathematics, reinforcement learning, and control used above.

← Back to writingBack to the top ↑