Batch Normalization

Batch normalization, or BatchNorm, is a neural network layer that normalizes each channel of its input with the mean and variance of the current mini-batch, then rescales the result with two learned parameters. Sergey Ioffe and Christian Szegedy introduced it in 2015, and it let their Inception image classifier reach the same accuracy in 14 times fewer training steps. BatchNorm allows higher learning rates, makes training less sensitive to weight initialization, and became a standard component of convolutional neural networks (CNNs) such as ResNet.

How Batch Normalization Works#

For a convolutional layer, BatchNorm computes one mean μ and one variance σ² per channel, pooling over every image in the batch and every spatial position. Each activation x then goes through two steps:

1. Normalize: x̂ = (x − μ) / √(σ² + ε), where a small ε, 1e-5 by default in PyTorch, prevents division by zero.
2. Scale and shift: y = γ·x̂ + β, where γ and β are learned per channel. They let the network restore whatever mean and spread suit the next layer, including undoing the normalization entirely.

BatchNorm usually sits between a convolution and its activation function. Because β already adds a per-channel offset, the preceding convolution's bias is redundant and is normally disabled.

Training vs. Inference#

During training, each forward pass also updates a running mean and running variance as exponential moving averages of the batch statistics. PyTorch's BatchNorm2d moves them 10% of the way toward each new batch by default (momentum=0.1). At inference, the layer normalizes with these running averages instead, so the prediction for one image no longer depends on which other images share its batch. model.train() and model.eval() switch between the two behaviors:

import torch

torch.manual_seed(0)
bn = torch.nn.BatchNorm2d(num_features=3)

# A batch of 8 three-channel feature maps with mean 5 and standard deviation 2
x = 5 + 2 * torch.randn(8, 3, 16, 16)

# Training mode normalizes with this batch's per-channel mean and variance
out = bn(x)
print(f"train mode: output mean {out.mean():.3f}, std {out.std():.3f}")

# The same call nudges the running averages 10% of the way toward the batch statistics
print(f"running mean {bn.running_mean[0]:.3f}, running var {bn.running_var[0]:.3f}")

# Eval mode normalizes with the running averages instead of the batch statistics
bn.eval()
out = bn(x)
print(f"eval mode:  output mean {out.mean():.3f}, std {out.std():.3f}")

In training mode, the output has mean 0.000 and standard deviation 1.000 even though the input has mean 5. After one batch, the running mean has only moved from 0 to 0.496 and the running variance from 1 to 1.294, so eval mode produces outputs with mean 3.920. Running averages need a few dozen batches to converge, and a model evaluated in the wrong mode, or before its statistics have settled, scores far below its true accuracy.

At inference, BatchNorm is a fixed per-channel linear transform, so it can be folded into the preceding convolution: the convolution weights are multiplied by γ/√(σ² + ε), and the bias becomes β − γμ/√(σ² + ε). The standard convolution block in YOLO26 is a bias-free convolution followed by BatchNorm and a SiLU activation, and Ultralytics fuses each of these BatchNorm layers into its convolution before prediction and export, which removes the layer while producing the same outputs.

Why Batch Normalization Helps#

The original paper attributed BatchNorm's benefit to reducing internal covariate shift, the change in each layer's input distribution as earlier layers update. Later experiments that injected random distribution shifts after BatchNorm layers found training barely affected, and traced the benefit to a smoother loss surface with more predictable gradients, which lets the optimizer take larger, more stable steps. Keeping pre-activations near zero mean and unit variance also keeps saturating activations out of their flat regions, which helps against vanishing gradients.

Because each example is normalized with the statistics of a randomly drawn batch, BatchNorm also adds noise that acts as a mild regularizer, and the original paper reported that it could remove the need for dropout in some cases.

Limitations#

• Small batches: Batch statistics get noisy when each device holds few images. At 2 images per GPU, Group Normalization, which computes statistics over groups of channels within each image, had 10.6% lower ImageNet error than BatchNorm on ResNet-50.
• Multi-GPU training: Each GPU normalizes with its own share of the batch unless layers are converted to SyncBatchNorm, which pools statistics across devices at the cost of extra communication.
• Sequence models: Variable-length sequences and batch-dependent statistics fit poorly, so transformers use layer normalization instead.

Batch Normalization vs. Related Terms#

• Data normalization: Rescales the raw inputs once during preprocessing, such as dividing pixel values by 255. BatchNorm normalizes intermediate activations at every layer and every training step.
• Layer normalization: Normalizes each example across its own features, so it behaves identically in training and inference and does not depend on batch size. RMSNorm, a simpler variant without mean subtraction, is common in large language models.
• Group normalization: Splits the channels of each image into groups and normalizes within each group, which suits detection and segmentation models trained with small batches.
• Instance normalization: Normalizes each channel of each image separately, removing per-image contrast. Swapping BatchNorm for instance normalization, at both training and test time, markedly improved the quality of fast neural style transfer.