Weight Decay

Weight decay is a regularization technique that pulls a model's learned weights toward zero during training. This repeated shrinkage discourages unnecessarily large parameter values and helps a neural network generalize to unseen examples. It is especially useful when a model has enough capacity to memorize noise in its training data.

Geoffrey Hinton introduced weight decay in his 1986 paper Learning Distributed Representations of Concepts. In its family-tree experiment, he reduced weight magnitudes by 0.2% after each update to make learned connections easier to interpret. Weight decay subsequently became widely used to control overfitting.

How Weight Decay Works#

During gradient descent, the optimizer changes parameters using gradients: derivatives that describe how each parameter affects the loss function. Weight decay adds a pull toward zero. Under the learning-rate-scaled convention used by many optimizers, a plain SGD update without momentum is new weight = (1 - learning rate × decay) × old weight - learning rate × gradient. This combines learning from prediction errors with proportional shrinkage.

The decay coefficient controls the strength of that pull. Zero disables it; larger positive values increase shrinkage at a fixed learning rate. The task gradient can still increase a weight when that improves predictions. Learning-rate decay progressively reduces the optimizer's step size; weight decay acts on parameter values.

Weight Decay vs. L2 Regularization#

L2 regularization adds a scaled sum of squared weights to the training objective. With plain stochastic gradient descent, differentiating this penalty produces the same proportional shrinkage, with appropriately matched coefficients. In linear regression, fitting with this squared-weight penalty is called ridge regression.

With the Adam optimizer, adding an L2 penalty sends the penalty gradient through Adam's moving averages and parameter-specific scaling. AdamW instead applies shrinkage separately from that adaptive gradient calculation. Ilya Loshchilov and Frank Hutter introduced AdamW in their 2017 paper Decoupled Weight Decay Regularization, published at ICLR 2019. This separation makes decay strength independent of Adam's gradient-history scaling.

Choosing the Decay Strength#

Weight decay is a hyperparameter, selected using validation data. Too little leaves overfitting insufficiently constrained. Excessive decay causes underfitting: the model fails to fit useful patterns, producing high errors on training and validation examples. Comparing validation scores across decay settings reveals the useful strength for a particular model and dataset.

The accumulated effect also depends on the learning-rate schedule and number of optimizer updates. Changing batch size or training epochs can change the number of decay applications; gradient accumulation combines multiple batches before an update. A dropout layer regularizes by randomly zeroing activations, while early stopping limits training duration.

Ultralytics YOLO exposes a weight_decay training setting, with a default of 0.0005, to control regularization when training models such as YOLO26. Its optimizer parameter groups exclude biases and normalization-layer parameters from weight decay.

Weight Decay in PyTorch#

The torch.optim.AdamW optimizer provides decoupled weight decay in PyTorch. Install the package with pip install torch; this example creates a zero task gradient so the printed update isolates weight decay:

import torch

weight = torch.nn.Parameter(torch.tensor([2.0]))
optimizer = torch.optim.AdamW([weight], lr=0.1, weight_decay=0.1)
before = weight.item()

# Create a zero gradient while keeping the parameter in the computation.
optimizer.zero_grad()
loss = (weight * 0).sum()
loss.backward()
optimizer.step()

print(f"Weight: {before:.2f} -> {weight.item():.2f}")

The output is Weight: 2.00 -> 1.98: one update multiplies the weight by 1 - 0.1 × 0.1, or 0.99, reducing its magnitude by 1%.