Latent Diffusion Model (LDM)

A latent diffusion model (LDM) is a diffusion model that generates data by denoising a compressed representation from a pretrained autoencoder instead of raw pixels. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer introduced the approach in High-Resolution Image Synthesis with Latent Diffusion Models, presented at CVPR 2022. The autoencoder strips out fine pixel detail that is costly to model and adds little to how an image looks, so the diffusion network spends its capacity on content and composition. Training and sampling therefore cost far less than diffusion in pixel space, which lets Stable Diffusion, the best-known LDM, generate images on a single consumer GPU. Most current image, video, and audio generators in generative AI follow the same two-stage design.

How Latent Diffusion Works#

The first stage compresses images. An encoder maps an image of size H×W×3 to a latent of size H/f×W/f×c, where f is the downsampling factor and c the number of latent channels, and a decoder maps latents back to pixels. Rombach and colleagues trained this autoencoder with a perceptual loss and a patch-based adversarial objective, as in a GAN discriminator, so reconstructions stay sharp, and regularized the latent either with a small KL penalty toward a standard normal distribution or with vector quantization. Downsampling factors from 4 to 16 balanced speed and quality: pixel-space training (f = 1) was slow, and f = 32 compressed so much that sample quality dropped. Stable Diffusion's autoencoder uses f = 8 and 4 channels, so this code encodes a photo, decodes it again, and measures what the round trip loses:

import urllib.request

import numpy as np
import torch
from diffusers import AutoencoderKL
from PIL import Image

# Stable Diffusion's autoencoder, the first stage of a latent diffusion model
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")

# Load a photo and scale its pixels to [-1, 1]
image = Image.open(urllib.request.urlopen("https://ultralytics.com/images/bus.jpg")).convert("RGB").resize((480, 640))
x = torch.tensor(np.array(image), dtype=torch.float32).permute(2, 0, 1)[None] / 127.5 - 1

# Encode to the latent that diffusion runs on, then decode back to pixels
with torch.no_grad():
    latent = vae.encode(x).latent_dist.mode()
    recon = vae.decode(latent).sample.clamp(-1, 1)

psnr = 10 * torch.log10(4 / torch.mean((recon - x) ** 2))
print(f"image {tuple(x.shape)} -> latent {tuple(latent.shape)}: {x.numel() // latent.numel()}x fewer values")
print(f"reconstruction PSNR: {psnr:.1f} dB")
Image.fromarray(((recon[0] + 1) * 127.5).byte().permute(1, 2, 0).numpy()).save("bus_reconstructed.jpg")

It prints image (1, 3, 640, 480) -> latent (1, 4, 80, 60): 48x fewer values and reconstruction PSNR: 23.0 dB. The saved bus_reconstructed.jpg keeps the bus, the people, and the street, while the small lettering on the side of the bus comes back garbled, the fine detail a 4-channel latent cannot hold.

The second stage is an ordinary diffusion model trained on these latents. Training encodes an image, adds Gaussian noise at a random step, and teaches a denoising network, a U-Net in the original work and a diffusion transformer in most newer models, to predict that noise, or in rectified flow models the velocity that carries the noise toward the clean latent. Conditions enter through cross-attention layers that let the latent attend to the embeddings of a text prompt, a class label, or a semantic map. Generation starts from random latent noise, denoises it step by step under the condition, and runs the decoder once at the end, so every expensive step works on the small latent.

From Stable Diffusion to Current Models#

• Stable Diffusion: Released in August 2022, it scaled the LDM recipe to 512×512 text-to-image generation with a 64×64×4 latent, an 860-million-parameter U-Net, and a CLIP text encoder, with weights anyone could download.
• Stable Diffusion 3: Moved to a 16-channel autoencoder, which raised reconstruction PSNR from 25.12 dB with 4 channels to 28.62 dB, and replaced the U-Net with a multimodal diffusion transformer trained with rectified flow. Black Forest Labs' FLUX.1 models also denoise 16-channel latents with a rectified flow transformer.
• Video and audio: Video generators compress clips in space and time, and OpenAI described Sora as compressing videos into a lower-dimensional latent space and splitting it into spacetime patches. AudioLDM and Stability AI's Stable Audio apply the same approach to spectrogram or waveform latents.
• Latent consistency models: Distill an LDM so it generates in 2 to 4 steps instead of 20 to 50, which makes near-real-time generation possible.

LDMs vs. Related Terms#

• Pixel-space diffusion models: Run the noising and denoising process on every pixel. An LDM is a diffusion model too, and differs only in where the process runs.
• Variational autoencoders: A VAE generates by decoding a sampled latent in one pass, and its samples tend to be blurry. An LDM uses a VAE-style autoencoder only for compression and leaves generation to the diffusion model.
• Stable Diffusion: One family of LDMs from Stability AI, CompVis, and Runway, so every Stable Diffusion model is an LDM while LDMs also include FLUX, video generators, and audio models.

Applications#

LDMs power text-to-image generation, image inpainting, where the original paper set a new state of the art at the time, super-resolution, and layout-to-image generation from semantic maps. In computer vision they also produce synthetic data: rare defects, unusual weather, or uncommon viewpoints that are hard to photograph. Generated images arrive without labels, and the Ultralytics auto_annotate function labels a folder of them by detecting objects with a YOLO26 model and outlining each detected box with a Meta SAM segmentation model, writing YOLO-format segmentation labels for training.