Stable Diffusion

Stable Diffusion is a family of open-weight text-to-image models from Stability AI that generate and edit images from natural-language prompts. The first version, released in August 2022, came from a collaboration with the CompVis group at LMU Munich and Runway and applied their latent diffusion research to 512×512 images from a subset of the LAION-5B dataset. Its 860-million-parameter U-Net and 123-million-parameter text encoder ran on a consumer GPU with at least 10 GB of memory, and because anyone could download the weights, it became the base for a large ecosystem of fine-tuned models, adapters, and editing tools. The current release, Stable Diffusion 3.5 from October 2024, replaces the U-Net with a multimodal diffusion transformer trained with rectified flow.

How Stable Diffusion Works#

Stable Diffusion is a diffusion model that denoises a compressed image instead of pixels, using three networks:

• Variational autoencoder (VAE): The encoder compresses a 512×512×3 image into a 64×64×4 latent, 48 times fewer values, and the decoder turns a finished latent back into an image. All denoising happens on the small latent, which is why the model fits on consumer hardware.
• Text encoder: Converts the prompt, up to 77 tokens, into embeddings. Stable Diffusion 1.x uses OpenAI's CLIP ViT-L/14, 2.x uses OpenCLIP ViT-H/14, SDXL combines two CLIP encoders, and Stable Diffusion 3 adds a T5-XXL language model to the two CLIP encoders for better prompt understanding.
• Denoising network: Predicts the noise to remove at each step. Versions 1.x, 2.x, and SDXL use a U-Net whose cross-attention layers let image features attend to the text embeddings, and versions 3 and 3.5 use an MMDiT that processes image and text tokens together.

Generation starts from a random 64×64×4 latent. A scheduler runs the denoising network for typically 20 to 50 steps, each time removing part of the predicted noise, and classifier-free guidance strengthens the prompt's influence. The VAE decoder then produces the image:

Because the starting latent is random, the same prompt with a different seed gives a different image, and the same seed and settings reproduce it.

Model Versions#

• Stable Diffusion 1.x (2022): 512×512 generation with an 860M-parameter U-Net; version 1.5 became a common base for community fine-tunes, and the ControlNet paper built on it. The weights use the CreativeML OpenRAIL-M license.
• Stable Diffusion 2.x (late 2022): Switched to the OpenCLIP text encoder and added a 768×768 model.
• SDXL (July 2023): Native 1024×1024 generation with a U-Net of about 2.6 billion parameters and two text encoders. SDXL Turbo, released that November, uses adversarial diffusion distillation to generate an image in a single step.
• Stable Diffusion 3 and 3.5 (2024): Replace the U-Net with a rectified flow transformer that has separate weights for image and text tokens. Stable Diffusion 3.5 Large has 8.1 billion parameters and 3.5 Medium has 2.5 billion. The Stability AI Community License allows free commercial use for organizations with less than $1 million in annual revenue.

Editing Images with Stable Diffusion#

• Image-to-image: Encodes an existing picture, noises it to an intermediate step set by a strength value, and denoises it with a new prompt, so the result keeps the original's layout and colors while changing its style or content.
• Fine-tuning: DreamBooth and LoRA adapt the model to a new subject or style from a few dozen images, and ControlNet adds edge, depth, or pose maps as structural conditions.
• Image inpainting: Regenerates only a masked region to match a prompt and its surroundings. The Stable Diffusion inpainting checkpoints add five input channels to the U-Net, four for the encoded masked image and one for the mask itself.

Inpainting masks are white where the image should change and black where it is kept. Ultralytics YOLO26 instance segmentation models produce such masks automatically, for example to replace or remove every vehicle of one class in a street photo:

import cv2
from ultralytics import YOLO

# Segment the bus (COCO class 5) with a YOLO26 instance segmentation model
model = YOLO("yolo26n-seg.pt")
results = model("https://ultralytics.com/images/bus.jpg", classes=[5], retina_masks=True)
result = results[0]

# White marks the pixels Stable Diffusion should repaint, black the pixels it keeps
mask = (result.masks.data.sum(dim=0) > 0).cpu().numpy().astype("uint8") * 255
cv2.imwrite("bus_inpaint_mask.png", mask)

The saved bus_inpaint_mask.png has the photo's full resolution because retina_masks=True returns masks at the original image size. Given the photo, this mask, and a prompt such as "a red double-decker bus," an inpainting pipeline regenerates the bus and keeps the street around it.

Real-World Applications#

• Synthetic data for vision models: Teams generate images of rare defects, unusual weather, or uncommon viewpoints to fill gaps in training datasets, and use inpainting to place new objects into real backgrounds.
• Concept art and design: Game studios, architects, and product designers explore many styles and compositions in minutes during early concept work.
• Marketing and e-commerce imagery: Image-to-image and ControlNet keep a product's shape fixed while the prompt changes its setting, lighting, and background.

Stable Diffusion vs. Related Models#

• Latent diffusion models: Stable Diffusion is the best-known implementation of the latent diffusion architecture, so every Stable Diffusion model is a latent diffusion model.
• GANs: Generate an image in one forward pass but train less stably and cover less variety. Stable Diffusion needs many denoising steps unless it is distilled, as in SDXL Turbo.
• Closed generators: Services such as OpenAI's image models and Midjourney run only on their providers' servers. Stable Diffusion weights can be downloaded, run offline, inspected, and fine-tuned.
• FLUX.1: Black Forest Labs, founded in 2024 by researchers behind the original latent diffusion paper, including Robin Rombach, Andreas Blattmann, and Patrick Esser, released the 12-billion-parameter FLUX.1 family, which uses the same rectified flow transformer approach as Stable Diffusion 3.