Backpropagation

Backpropagation, short for "backward propagation of errors," is the algorithm that computes how much each weight in a neural network contributed to the error of a prediction. Starting from the loss function, it applies the chain rule layer by layer back toward the input and reuses each intermediate result, so one backward sweep yields the gradient for every weight, whether the model has thousands of parameters or billions. An optimizer then uses those gradients to update the weights, which makes backpropagation the learning step behind virtually every deep learning model. Rumelhart, Hinton, and Williams popularized it for neural networks in the 1986 Nature paper Learning representations by back-propagating errors.

How Backpropagation Works#

Training on a batch takes two passes. The forward pass feeds the inputs through each layer, stores the intermediate activations, and ends with the loss. The backward pass starts at the loss and visits the layers in reverse order. Each layer receives the gradient of the loss with respect to its output, called the upstream gradient, and multiplies it by its own local derivatives to produce two results: the gradient for the layer's weights, which is kept for the optimizer, and the gradient with respect to the layer's input, which becomes the upstream gradient for the layer below.

Because every layer needs only its upstream gradient and its stored activations, no derivative is ever computed twice. In automatic differentiation terms, backpropagation is reverse-mode differentiation of the network's computational graph, and PyTorch builds that graph during the forward pass and walks it backward when loss.backward() is called. In a two-weight network, PyTorch's autograd result can be checked against the chain rule written out by hand:

import torch

# A two-weight network: prediction = w2 * tanh(w1 * x)
x, target = torch.tensor(1.5), torch.tensor(0.5)
w1 = torch.tensor(0.8, requires_grad=True)
w2 = torch.tensor(-1.2, requires_grad=True)

# Forward pass, keeping the hidden activation h for the backward pass
h = torch.tanh(w1 * x)
loss = (w2 * h - target) ** 2

# Backward pass: autograd applies the chain rule from the loss back to each weight
loss.backward()

# The same chain rule by hand, reusing the upstream gradient dL/dy for both weights
dL_dy = 2 * (w2 * h - target)
dL_dw2 = dL_dy * h
dL_dw1 = dL_dy * w2 * (1 - h**2) * x
print(f"autograd: dL/dw1 = {w1.grad:.4f}, dL/dw2 = {w2.grad:.4f}")
print(f"by hand:  dL/dw1 = {dL_dw1:.4f}, dL/dw2 = {dL_dw2:.4f}")

Both lines print dL/dw1 = 1.6475, dL/dw2 = -2.5016. The hand-written version computes dL_dy once and reuses it for both weights, the same reuse that lets autograd handle networks with billions of weights. Backpropagation through time (BPTT) is this same algorithm applied to a recurrent neural network unrolled across its time steps.

Backpropagation vs. the Optimizer#

Backpropagation only computes gradients and changes no weights. An optimization algorithm such as stochastic gradient descent (SGD) or Adam turns those gradients into weight updates, scaled by the learning rate. PyTorch adds each backward pass's result to the gradients already stored on the weights, so one optimizer step can combine several batches:

Ultralytics YOLO training uses this gradient accumulation to reach a nominal batch of 64 images (nbs=64): after warmup, a batch=16 run backpropagates every batch but steps the optimizer every fourth batch, after clipping the combined gradient to a norm of 10.

Vanishing and Exploding Gradients#

The backward pass multiplies local derivatives layer after layer, so the gradient's size can shrink or grow geometrically with depth. When most factors are below 1, as with saturated sigmoid units, the earliest layers receive almost no signal, which is the vanishing gradient problem. When most factors are above 1, the exploding gradient problem produces huge updates and NaN losses. ReLU-family activations, scaled weight initialization, batch normalization, and the skip connections of residual networks keep gradient magnitudes stable across hundreds of layers.

Compute and Memory Cost#

Every activation that a local derivative needs must stay in memory from the forward pass until the backward pass reaches it, so training memory grows with batch size, input resolution, and depth, and training needs far more memory than inference. Gradient checkpointing stores only some activations and recomputes the rest during the backward pass, trading extra compute for lower memory, and mixed precision keeps most activations in 16-bit formats that take half the space of 32-bit floats. In compute, the backward pass costs about twice the forward pass because each layer computes gradients for both its weights and its inputs, which is why scaling-law estimates count about 6 floating-point operations per parameter per training token: 2 in the forward pass and 4 in the backward pass.