Diffusion Language Models

A diffusion language model generates text by starting from a fully corrupted sequence, usually all mask tokens, and refining many positions in parallel over a series of denoising steps, where a conventional large language model (LLM) writes one token at a time from left to right. It carries the idea behind image diffusion models over to discrete text: a forward process gradually destroys a sequence, and a transformer learns to reverse it. The approach reached production scale in 2025. LLaDA, an 8-billion-parameter diffusion model trained from scratch, was competitive with LLaMA3 8B on in-context learning; Inception Labs' Mercury Coder models ran at 1,109 and 737 tokens per second on NVIDIA H100 GPUs in independent tests; and Google DeepMind's experimental Gemini Diffusion lists an average sampling speed of 1,479 tokens per second.

How Masked Diffusion Works#

LLaDA and most other recent models use masked, or absorbing-state, diffusion. The forward process picks a noise level t between 0 and 1 and replaces each token with a mask symbol with probability t:

import torch

torch.manual_seed(6)
tokens = ["the", "cat", "sat", "on", "the", "warm", "mat", "by", "the", "door"]

# Forward process of a masked diffusion LM: at noise level t, each token is masked with probability t
for t in [0.25, 0.5, 0.75, 1.0]:
    masked = torch.rand(len(tokens)) < t
    noisy = ["[M]" if m else token for token, m in zip(tokens, masked)]
    print(f"t={t:.2f}: {' '.join(noisy)}")

The run prints the sentence with 2, 5, 8, and finally all 10 words masked. During training, a transformer without a causal mask sees one such sequence and predicts every masked token at once, with a cross-entropy loss function on the masked positions that LLaDA weights by 1/t. The objective generalizes BERT's masked language modeling, which always predicts 15% of tokens, to every masking ratio, and it bounds the model's log-likelihood so the result is a full generative model.

Generation runs the process backward from t = 1. Each step predicts all masked positions, commits the most confident predictions, and remasks the rest, a strategy LLaDA calls low-confidence remasking. Long outputs are often produced block by block, an approach formalized as block diffusion, so earlier blocks can be cached and the output length need not be fixed in advance:

From D3PM to Production Models#

Google's D3PM defined diffusion over discrete states in 2021, including the absorbing mask state most models now use. Stanford's Diffusion-LM diffused continuous word embeddings in 2022 to steer generated text toward syntax and topic constraints. Aaron Lou, Chenlin Meng, and Stefano Ermon's SEDD, which won an ICML 2024 Best Paper award, cut the perplexity of earlier text diffusion models by 25% to 75% and outperformed GPT-2. LLaDA then showed that the approach scales to billions of parameters and, because it reads context in both directions, surpassed GPT-4o on a reversal poem completion task that tests the reversal curse of left-to-right models. Inception serves Mercury through an OpenAI-compatible Chat Completions API, so the Ultralytics LLM interface can call it with api="chat.completions" and Inception's base_url, for example to turn YOLO26 detections into a written report.

Strengths and Trade-Offs#

Parallel unmasking needs fewer forward passes than there are tokens, which is the source of the speed, and quality falls as fewer steps commit more tokens each. Tokens committed in the same step are sampled independently, so committing many at once can produce combinations that do not fit together. Each step also attends over the whole block in both directions, which makes a step more expensive than one autoregressive step with a KV cache. Bidirectional context makes infilling and editing natural: the model can fill a gap in the middle of a function or a document using the text on both sides, which suits code completion and agentic coding tools that edit existing files.

Diffusion LMs vs. Related Terms#

• Autoregressive text generation: Predicts each token from the tokens to its left, one forward pass per token.
• Image diffusion: Stable Diffusion and other latent diffusion models add Gaussian noise to continuous pixels or latents, often with a diffusion transformer (DiT) as the denoiser.
• Diffusion forcing: Gives each token its own noise level, mixing next-token prediction with full-sequence diffusion, mainly for video and planning.
• Speculative decoding: Speeds up an autoregressive model by drafting several tokens with a smaller model and verifying them in one pass of the large model, which leaves its output distribution unchanged.