Diffusion Forcing

Diffusion forcing is a training method for sequence generators in which every token of a sequence, such as a video frame or a robot action, receives its own independently sampled noise level, and a causal model learns to denoise each token given a history of partly noisy ones. One model trained this way can roll a sequence out step by step like a large language model (LLM) predicting its next token and also steer the whole future toward a goal like a full-sequence diffusion model. Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann at MIT introduced it in July 2024 in Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, which appeared at NeurIPS 2024.

Noise as Partial Masking#

Teacher forcing trains an autoregressive model to predict the next token from a clean, ground-truth history, so at generation time the model meets its own imperfect outputs for the first time, and long video generation rollouts drift as errors compound. Full-sequence diffusion denoises every frame together at one noise level, which supports guidance but fixes the sequence length and cannot react to new observations partway through.

Diffusion forcing treats noise as partial masking: a clean token is fully visible, pure noise hides it completely, and every level in between hides part of it. Training with independent noise levels per token teaches one model every combination at once, including a clean past with a fully noisy next frame, which is next-token prediction, and equal noise on every frame, which is full-sequence diffusion. The paper proves that this objective optimizes a variational lower bound on the likelihood of all subsequences. Its reference implementation, Causal Diffusion Forcing, used a recurrent neural network whose hidden state summarizes the noisy history, and the authors later released a transformer version.

Sampling Schedules#

Because any mix of noise levels is in-distribution, generation can follow a two-dimensional schedule over denoising passes and frames. The paper's zig-zag schedule starts each frame's denoising a little later than the frame before it:

import torch

torch.manual_seed(0)
frames, levels = 5, 4  # a 5-frame clip; noise levels run from 0 (clean) to 4 (pure noise)

# Training: every frame gets its own random noise level, so any mix of clean and noisy frames is seen
print("Training noise levels:", torch.randint(0, levels + 1, (frames,)).tolist())

# Sampling with a zig-zag schedule: each row is one denoising pass over all frames, and frame i
# starts denoising i passes after frame 0, so the near future is always cleaner than the far future
for step in range(levels + frames):
    row = [min(max(levels - step + i, 0), levels) for i in range(frames)]
    print(f"pass {step}: {row}")

The training line draws levels such as [4, 4, 3, 0, 3] for one clip, a mix that neither teacher forcing nor full-sequence diffusion trains on. In the schedule, row 0 is pure noise, and by pass 4 the first frame is clean while the fifth is still fully noisy; the clip is finished after 8 passes. The paper uses this shape for planning because it treats the immediate future as more certain than the distant future.

Long rollouts use a second trick. Conditioning each new frame on slightly noisy copies of the frames already generated, instead of the clean copies, makes the model tolerant of its own small errors. On the DMLab navigation dataset, a model trained on 36-frame clips rolled out for 2,000 frames without diverging, where the baselines diverged.

Guidance, Planning, and Robotics#

Full-sequence guidance carries over: a reward or goal can steer the noisy future frames during sampling. The paper's Monte Carlo Guidance draws several possible futures and averages their guidance gradients, which improved results on maze planning tasks used in reinforcement learning research. For robotics, the authors used imitation learning to train a real robot arm to swap two fruits using a third slot, a task that needs memory because the fruits start in random slots. Diffusion forcing succeeded 80% of the time where the memoryless Diffusion Policy failed, and with visual distractions and even a fully covered camera it still reached 76%, against 48% for a next-frame diffusion baseline that treated the corrupted frames as ground truth. The same model also handled time series forecasting.

Diffusion Forcing in Video Models#

The MIT group's Diffusion Forcing Transformer (DFoT), from History-Guided Video Diffusion in February 2025, conditions on any number of past frames and uses that history as a guidance signal. Skywork AI's SkyReels-V2, released in April 2025, fine-tunes a pretrained full-sequence video model into a diffusion forcing model whose noise levels never decrease from one frame to the next, so videos can be extended chunk by chunk. Video world models, which predict how a scene evolves under an AI agent's actions, need the same kind of stable causal rollout. Whether a long rollout keeps objects consistent can be measured with object tracking: Ultralytics YOLO26 in track mode assigns each detected object an ID that persists across frames, so a car that keeps one track ID through a generated clip has kept its identity.

Diffusion Forcing vs. Related Terms#

• Diffusion language models: Unmask discrete text tokens in parallel over several steps, with one shared masking ratio per step.
• Diffusion policies: Denoise a chunk of robot actions with the same noise level on every action, conditioned on clean observations.
• Text-to-video diffusion: Most systems generate a fixed-length clip with full-sequence diffusion, and diffusion forcing variants extend clips beyond that length one chunk at a time.