Vanishing Gradient

The vanishing gradient problem occurs when the gradients that backpropagation sends from the loss toward the first layers of a deep neural network shrink geometrically with depth, so the earliest layers receive updates too small to learn anything. Because the gradient at a layer is a product of local derivatives from every layer above it, a network whose layers each scale the gradient by less than 1 can lose many orders of magnitude between output and input. Sepp Hochreiter analyzed the effect in his 1991 diploma thesis, and it kept deep feedforward networks and recurrent neural networks (RNNs) hard to train until better activations, initializations, and architectures arrived.

Why Gradients Vanish#

Backpropagation applies the chain rule, so the gradient reaching a layer is the upstream gradient multiplied by each later layer's weight matrix and activation function derivative. The sigmoid derivative never exceeds 0.25, its value at an input of 0, and falls toward 0 when the unit saturates at large positive or negative inputs. The tanh derivative peaks at 1 and also falls toward 0 when saturated. Twenty sigmoid layers therefore multiply the gradient by at most 0.25²⁰, below 10⁻¹², before the weights are even counted. Weight scale matters as much as the activation: Glorot and Bengio traced much of the difficulty to initialization and proposed weight variances scaled to each layer's number of inputs and outputs, so that activation and gradient variances stay roughly constant across layers.

The effect is easy to measure by comparing weight-gradient sizes at the top and bottom of a 20-layer sigmoid network, with and without residual shortcuts:

import torch

torch.manual_seed(0)
x = torch.randn(64, 32)

# The same 20-layer sigmoid network, without and with residual shortcuts
for residual in (False, True):
    layers = [torch.nn.Linear(32, 32) for _ in range(20)]
    h = x
    for layer in layers:
        out = torch.sigmoid(layer(h))
        h = h + out if residual else out
    h.sum().backward()

    # Compare the weight-gradient size at the last layer and the first layer
    last, first = layers[-1].weight.grad.norm(), layers[0].weight.grad.norm()
    print(f"residual={residual}: layer 20 gradient {last:.1e}, layer 1 gradient {first:.1e}")

Without shortcuts, the last layer's gradient norm is 2.5e+02 and the first layer's is 4.2e-15, about 17 orders of magnitude smaller, so the first layer's weights effectively never change. With residual shortcuts, the first layer's gradient is 5.6e+01, about 38 times smaller than the last layer's 2.1e+03.

Vanishing vs. Exploding Gradients#

Both problems come from the same repeated multiplication. Vanishing gradients stall learning silently: the loss still falls, driven by the last layers, while the early layers stay close to their random initialization. Exploding gradients grow geometrically instead and show up loudly as loss spikes and NaN values. Gradient clipping caps an exploding gradient's norm before each update and has no effect on a gradient that has already vanished. The dying ReLU problem is a related failure in which a ReLU unit's input stays negative, so its gradient is exactly 0 and the unit stops learning.

How to Prevent Vanishing Gradients#

The most widely used fix is the residual connection, the identity shortcut at the heart of residual networks (ResNets). A residual block outputs x + F(x), so its gradient with respect to x is 1 plus the derivative of F, and the shortcut path carries the upstream gradient through unchanged.

Residual shortcuts let ResNets train with more than 100 layers, and they appear in nearly every modern deep architecture, from transformers to YOLO26, whose C3k2 blocks wrap residual bottlenecks and whose standard convolution blocks pair batch normalization with SiLU activations.

Other fixes work alongside residual connections:

• Non-saturating activations: ReLU has a derivative of exactly 1 for positive inputs, and Leaky ReLU and SiLU keep a nonzero gradient for negative inputs as well.
• Scaled initialization: Xavier (Glorot) initialization suits tanh and sigmoid layers, and He initialization doubles the weight variance to compensate for ReLU zeroing half of its inputs.
• Normalization layers: Batch normalization and layer normalization keep pre-activations near zero mean and unit variance, out of the flat regions of saturating activations.
• Gated recurrent architectures: Long short-term memory (LSTM) networks and gated recurrent units (GRUs) carry their state forward through gated additive updates instead of repeated multiplication, which let the original LSTM bridge time lags of more than 1,000 steps.