Exploding Gradient

An exploding gradient is a training failure in which the gradients computed by backpropagation grow exponentially as they pass backward through a deep or recurrent neural network. Gradient descent then applies enormous updates to the model weights, the loss function spikes or turns into NaN or inf, and training diverges. It is the mirror image of the vanishing gradient problem. Pascanu, Mikolov, and Bengio analyzed both in recurrent neural networks and proposed gradient norm clipping for exploding gradients, which remains the standard safeguard in deep learning training today.

Why Gradients Explode#

By the chain rule, the gradient that reaches an early layer is a product of one factor per later layer, or per time step in a recurrent network. When those factors are consistently larger than 1, the product grows exponentially with depth: a factor of 1.5 repeated over 30 layers multiplies the signal by about 190,000. Common triggers include poor weight initialization, a learning rate that is too high, unnormalized inputs, long sequences in recurrent models, and overflow in mixed precision training, where FP16 values above 65,504 become inf. The signs are a loss curve that suddenly jumps or turns NaN, gradient-norm logs that rise by orders of magnitude between steps, and weights that become NaN.

Prevention and Mitigation#

Most modern training pipelines combine several of these defenses.

• Gradient Clipping: When the global L2 norm of all gradients exceeds a threshold, every gradient is scaled down by the same factor. The update keeps its direction and only its length is capped.
• Batch Normalization: Batch normalization and layer normalization rescale activations at every layer, which keeps signals at a stable scale from layer to layer and smooths the loss landscape.
• Careful Initialization: Xavier (Glorot) and He initialization scale the starting weights so activation variance stays roughly constant from layer to layer.
• Residual Connections: Skip connections give gradients an identity path around each block, so the backward signal is a sum of paths rather than one long product.
• Learning Rate Warmup: Starting with a small learning rate and ramping it up over the first steps avoids huge early updates while weights and optimizer statistics are still unsettled.

The snippet below builds a 30-layer stack whose layers each amplify their input by 1.5x, then clips the resulting gradient with PyTorch:

import torch
from torch import nn

torch.manual_seed(0)

# 30 stacked linear layers, each initialized to amplify its input by 1.5x
layers = [nn.Linear(64, 64, bias=False) for _ in range(30)]
for layer in layers:
    nn.init.orthogonal_(layer.weight, gain=1.5)
model = nn.Sequential(*layers)

loss = model(torch.randn(8, 64)).pow(2).mean()
loss.backward()

# clip_grad_norm_ rescales all gradients in place and returns their norm before clipping
norm_before = nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
norm_after = nn.utils.get_total_norm([p.grad for p in model.parameters()])
print(f"Gradient norm before clipping: {norm_before:.2e}, after: {norm_after:.2f}")

The raw gradient norm is 9.90e+10, and after clipping it is 1.00, so the next optimizer step stays bounded however large the raw gradient was.

Ultralytics YOLO26 training runs this safeguard on every step: the trainer unscales the mixed precision gradients, clips their global norm to 10.0, and lets the gradient scaler skip any update that still contains inf or NaN values. A custom trainer can change the limit, such as tightening it to 0.1 for RT-DETR.

Exploding vs. Vanishing Gradients#

Both problems come from the same product of per-layer factors in backpropagation. Factors above 1 make gradients explode, producing unstable updates, numerical overflow, and divergence, and the usual fix is clipping. Factors below 1 make gradients shrink toward zero, so early layers stop learning, and the usual fixes are non-saturating activations such as ReLU, residual connections, and gated recurrent units such as LSTMs. Normalization and careful initialization help against both.

Real-World Applications#

Large language models train for hundreds of thousands of steps on long text sequences, where sudden loss spikes are a known hazard, and their training recipes clip gradients: GPT-3 was trained with Adam and a global gradient norm clipped at 1.0.

In computer vision, deep object detection networks train on COCO-scale datasets at high learning rates with mixed precision, so clipping and loss scaling keep a rare extreme batch from corrupting the weights. Transformer-based detectors are more sensitive because gradients through their decoder cross-attention can spike, so DETR-family models such as RT-DETR and DINO typically clip at a max norm of 0.1.