Normalizing Flows

A normalizing flow is a generative AI model that transforms a simple base distribution, usually a standard Gaussian, into a complex data distribution through a sequence of invertible, differentiable functions. Because every layer can run backward and its effect on volume is computed exactly, one flow does two jobs that most generative models cannot do together: it draws new samples by passing noise forward through the layers, and it computes the exact likelihood of an observed data point by passing it backward to the base distribution. Esteban Tabak and colleagues formulated flows for density estimation in 2010 and 2013, and Danilo Rezende and Shakir Mohamed of DeepMind brought the name into deep learning with Variational Inference with Normalizing Flows at ICML 2015. "Normalizing" refers to turning the data distribution into a normal, or Gaussian, one, and is unrelated to batch normalization.

The Change-of-Variables Formula#

If an invertible function f maps a data point x to a latent point z = f(x), the data's log-likelihood is log p(x) = log p_z(f(x)) + log |det J|, where J is the Jacobian matrix of f and the determinant measures how much f stretches or squeezes volume around x. For a general function of D variables, that determinant costs on the order of D³ operations, far too slow for images with hundreds of thousands of dimensions. Flow layers are therefore designed with triangular Jacobians, whose determinant is just the product of the diagonal entries. Stacking many such layers composes their transformations and adds their log-determinants, and training maximizes the log-likelihood of the training data directly.

The affine coupling layer from RealNVP is the classic design. Half of the input passes through unchanged and feeds a small neural network that sets a scale and shift for the other half, as in this PyTorch layer:

import torch
from torch import nn

torch.manual_seed(0)
# Affine coupling layer (RealNVP): x1 passes through unchanged and sets the scale and shift for x2
net = nn.Sequential(nn.Linear(2, 32), nn.Tanh(), nn.Linear(32, 4))


def coupling(x):
    """Map data x to z and return log|det J|, which is just the sum of the log-scales."""
    x1, x2 = x.chunk(2, dim=-1)
    log_s, t = net(x1).chunk(2, dim=-1)
    return torch.cat([x1, x2 * log_s.exp() + t], dim=-1), log_s.sum(dim=-1)


x = torch.randn(4)
z, log_det = coupling(x)
jacobian = torch.autograd.functional.jacobian(lambda v: coupling(v)[0], x)
print(f"log|det J|: {log_det:.4f} from the formula, {torch.logdet(jacobian):.4f} from the full 4x4 Jacobian")

# Inverse: z1 equals x1, so the same scale and shift can be recomputed and undone
z1, z2 = z.chunk(2)
log_s, t = net(z1).chunk(2)
x_rec = torch.cat([z1, (z2 - t) * (-log_s).exp()])
print(f"Largest reconstruction error: {(x_rec - x).abs().max():.1e}")

The cheap formula and the full Jacobian both give a log-determinant of 0.1489, and the inverse recovers the input to within 1.2e-07, float32 rounding. The network inside the layer never has to be inverted, so it can be arbitrarily complex.

Flow Architectures#

• Coupling flows: Laurent Dinh and colleagues' NICE (2014) and RealNVP (2016) alternate which half is transformed, and OpenAI's Glow (2018) added learned invertible 1 x 1 convolutions and generated realistic 256 x 256 faces. Sampling and density evaluation are both fast.
• Autoregressive flows: Transform each dimension conditioned on the ones before it. Masked autoregressive flows evaluate densities in one pass but sample one dimension at a time, and inverse autoregressive flows do the reverse. Apple's TarFlow (2024) stacks autoregressive transformer blocks over image patches and was the first standalone flow whose samples rivaled diffusion models in quality.
• Continuous flows: Define the transformation as the solution of an ordinary differential equation, which removes the layer-design constraints; FFJORD (2018) tracks the change in log-density with a stochastic trace estimator.

Applications#

• Anomaly detection: DifferNet (2021), CFLOW-AD, and FastFlow fit flows to the features of a pretrained convolutional neural network on images of defect-free products and flag low-likelihood images as defective. FastFlow reported 99.4% AUC for anomaly detection on the MVTec AD industrial inspection benchmark in 2021.
• Speech synthesis: NVIDIA's WaveGlow (2018) generated speech audio from mel spectrograms in a single parallel pass.
• Scientific sampling: Frank Noé and colleagues' Boltzmann generators (2019) sample equilibrium states of molecules and proteins, using the exact likelihood to reweight samples to the true physical distribution.

These detectors model features instead of pixels for a reason Eric Nalisnick and colleagues demonstrated in 2019: a Glow model trained on CIFAR-10 photos assigns higher likelihood to SVHN house-number images it has never seen, a failure later traced to flows modeling local pixel correlations more than image content. For defect detection, the input to the flow can be embeddings from any pretrained backbone: Ultralytics YOLO26's embed method returns one average-pooled feature vector per image, by default from the layer just before the detection head.

Normalizing Flows vs. Related Terms#

• Flow matching and rectified flow: Train a continuous flow by regressing its velocity field, with no log-determinant during training, and now power large image generators.
• Variational autoencoders and diffusion models: Train on a lower bound of the likelihood.
• GANs: Learn to fool a discriminator and give no likelihood at all.
• Generative flow networks (GFlowNets): Share the word "flow" but build discrete objects step by step with probability proportional to a reward.