Diffusion Models

Diffusion models are generative AI models that create data by learning to reverse a gradual noising process. During training, images or other data are corrupted with Gaussian noise over many small steps, and a neural network learns to predict the noise at each step. To generate, the model starts from pure random noise and removes it step by step until a new sample emerges. Jascha Sohl-Dickstein and colleagues proposed the approach in 2015 in Deep Unsupervised Learning using Nonequilibrium Thermodynamics, and Jonathan Ho, Ajay Jain, and Pieter Abbeel made it competitive in 2020 with Denoising Diffusion Probabilistic Models (DDPM), which set a state-of-the-art FID of 3.17 on CIFAR-10. In 2021 OpenAI researchers showed that diffusion models beat GANs on ImageNet image synthesis, and diffusion models and their flow matching relatives now power text-to-image systems such as Stable Diffusion, video generators, image editors, and scientific models.

How Diffusion Models Work#

The forward process is fixed rather than learned. It adds a little Gaussian noise at each of T steps; DDPM uses T = 1,000 with a noise variance that rises linearly from 0.0001 to 0.02 per step. Because a sum of Gaussian noise is still Gaussian, a training image can jump straight to any step t in closed form, xt = sqrt(alpha_bar_t) * x0 + sqrt(1 - alpha_bar_t) * noise, where alpha_bar_t is the product of 1 - beta over the first t steps. Training repeatedly picks an image, a random step, and fresh noise, builds xt, and teaches the network to predict that noise with a mean squared error, the simplified objective from DDPM. This code applies the forward process to a photo with DDPM's schedule:

import urllib.request

import numpy as np
import torch
from PIL import Image

# DDPM's linear schedule: 1,000 steps whose noise variance rises from 0.0001 to 0.02
betas = torch.linspace(1e-4, 0.02, 1000)
alpha_bar = torch.cumprod(1 - betas, dim=0)

# Load a photo and scale its pixels to [-1, 1]
image = Image.open(urllib.request.urlopen("https://ultralytics.com/images/bus.jpg")).convert("RGB").resize((256, 256))
x0 = torch.tensor(np.array(image), dtype=torch.float32).permute(2, 0, 1) / 127.5 - 1

# Jump straight to step t in closed form: x_t = sqrt(alpha_bar) * x0 + sqrt(1 - alpha_bar) * noise
frames = []
for t in (0, 99, 299, 599, 999):
    xt = alpha_bar[t].sqrt() * x0 + (1 - alpha_bar[t]).sqrt() * torch.randn_like(x0)
    frames.append(((xt.clamp(-1, 1) + 1) * 127.5).byte().permute(1, 2, 0).numpy())
    print(f"step {t + 1}: image weight {alpha_bar[t].sqrt():.3f}, noise weight {(1 - alpha_bar[t]).sqrt():.3f}")
Image.fromarray(np.hstack(frames)).save("forward_diffusion.jpg")

The image weight falls from 1.000 at step 1 to 0.947 at step 100, 0.630 at step 300, 0.161 at step 600, and 0.006 at step 1,000, while the noise weight rises to 1.000. The saved forward_diffusion.jpg shows the bus sharp, then grainy, then faint, then gone.

Generation runs the process backward. It starts from pure Gaussian noise, and at every step the network predicts the noise in the current sample, a solver removes part of it, and the slightly cleaner result feeds the next step. A text prompt or class label enters the network as a condition, often through cross-attention. Classifier-free guidance, introduced by Jonathan Ho and Tim Salimans, trains one network both with and without the condition and extrapolates from the unconditional prediction toward the conditional one at sampling time; a higher guidance scale, such as the default 7.5 in Stable Diffusion 1.x pipelines, follows the prompt more closely at the cost of diversity.

Making Sampling Faster#

DDPM sampling takes 1,000 network evaluations per image, so much of the field's progress has gone into cutting that number:

• Better samplers: DDIM reuses a trained DDPM with a deterministic sampler and produces high-quality samples 10 to 50 times faster in wall-clock time. Later ODE solvers bring typical generation down to about 20 to 50 steps.
• Latent diffusion: A latent diffusion model denoises a compressed representation from an autoencoder instead of pixels. Stable Diffusion's autoencoder turns a 512×512×3 image into a 64×64×4 latent, 48 times fewer values.
• Distillation: Consistency models learn to map any noisy sample directly to a clean one, enabling one-step generation, and latent consistency models bring the same idea to Stable Diffusion with 2 to 4 steps.
• Straighter paths: Flow matching and rectified flow train along straight noise-to-data paths that solvers can follow in fewer steps.

The denoising network itself has moved from the convolutional U-Net of DDPM and Stable Diffusion 1.x to the diffusion transformer (DiT) used by most current image and video generators.

Diffusion Models vs. Other Generative Models#

• Generative adversarial networks (GANs): A GAN generates a sample in one forward pass, but its adversarial training between generator and discriminator is prone to instability and mode collapse, where the generator covers only part of the data. Diffusion training is a stable regression and covers the data distribution more completely, at the cost of many sampling steps.
• Variational autoencoders (VAEs): A VAE decodes a latent vector into a sample in one pass, and its samples tend to be blurrier. Latent diffusion models use a VAE-style autoencoder only to compress images and leave generation to the diffusion model.
• Autoregressive models: Language models and visual autoregressive models generate one token or scale at a time, conditioned on what came before. A diffusion model refines every part of the sample in parallel at each step, and diffusion language models apply the same denoising to text.

Real-World Applications#

• Image generation and editing: Text-to-image models create images from prompts, image inpainting regenerates a masked region to match its surroundings, ControlNet constrains layout with edge, depth, or pose maps, and diffusion upscalers perform super-resolution.
• Video generation: Video generation models denoise compressed latents that span many frames at once, so each frame is generated together with its neighbors.
• Synthetic training data: Generated images fill gaps in vision datasets, such as rare defects, unusual weather, or uncommon viewpoints. In a dataset on Ultralytics Platform, the Generate Similar Images action creates 1 to 16 new images per batch inspired by an existing one, and the images you keep join the dataset unlabeled, ready for annotation.
• Robotics: Diffusion policies generate robot action sequences by denoising them, conditioned on camera images.
• Science: AlphaFold 3 predicts the structures of complexes of proteins, nucleic acids, and small molecules with a diffusion module that denoises raw atom coordinates, replacing the structure module of AlphaFold 2.