Regularization

Regularization is any change to how a model is trained that is meant to lower its error on new data, even if its error on the training data rises. It counters overfitting, in which a model with enough capacity memorizes noise and quirks of the examples it saw. Regularization limits a model's effective complexity by penalizing large weights, injecting noise, adding varied data, or stopping training early. In bias-variance terms, it accepts a little extra bias in exchange for a larger drop in variance.

L1 and L2 Penalties#

The classic form adds a penalty on the weights to the loss function: total loss = data loss + λ × penalty. L2 regularization penalizes the sum of squared weights, Σw², which pulls every weight toward zero in proportion to its size, and in linear models it is called ridge regression. L1 regularization penalizes the sum of absolute weight values, whose constant pull drives many weights exactly to zero, so L1-penalized linear regression, the lasso, doubles as feature selection. Elastic net combines both. The penalty strength λ, called alpha in scikit-learn, is a hyperparameter: too small and the model still overfits, too large and it underfits. A degree-9 polynomial fit to 12 noisy points from a sine wave shows both failure modes:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)

# 12 noisy training points from a sine wave, plus clean test points
x_train = rng.uniform(0, 1, (12, 1))
y_train = np.sin(2 * np.pi * x_train[:, 0]) + rng.normal(0, 0.2, 12)
x_test = np.linspace(0, 1, 200)[:, None]
y_test = np.sin(2 * np.pi * x_test[:, 0])

# The same degree-9 polynomial, fit with a growing L2 penalty on its coefficients
for alpha in (0.0, 1e-4, 1e-2, 1.0):
    model = make_pipeline(PolynomialFeatures(9), Ridge(alpha=alpha)).fit(x_train, y_train)
    train_mse = np.mean((model.predict(x_train) - y_train) ** 2)
    test_mse = np.mean((model.predict(x_test) - y_test) ** 2)
    weight_norm = np.linalg.norm(model[-1].coef_)
    print(f"alpha {alpha:g}: train MSE {train_mse:.4f}, test MSE {test_mse:.4f}, |w| {weight_norm:.1f}")

Without a penalty, the polynomial bends through the noise: its coefficient norm reaches about 425,000, its training error is 0.0148, and its test error is 0.1859. A penalty of 0.0001 shrinks the coefficient norm to 20.8 and cuts test error elevenfold to 0.0170, while training error rises to 0.0334. At an alpha of 1, the penalty dominates and the model underfits, with a test error of 0.3998.

Weight Decay and Dropout in Neural Networks#

In neural networks, L2 regularization usually takes the form of weight decay: every update also shrinks each weight by a small fraction of its value. For plain stochastic gradient descent, the two are equivalent. For adaptive optimizers such as Adam, an L2 term added to the gradient is divided by Adam's per-parameter scaling, so weights with large gradients are barely regularized. Decoupled weight decay applies the shrinkage directly instead, and the resulting AdamW optimizer is the standard choice for transformers.

A dropout layer zeroes each unit's output with probability p at every training step and scales the surviving outputs by 1/(1 − p), so the network cannot depend on any single unit and in effect trains many thinned subnetworks that share weights. At inference, every unit stays active. A drop probability of 0.5 for hidden units was close to optimal across a wide range of networks and tasks in the original experiments.

Ultralytics YOLO training exposes both: weight_decay, 0.0005 by default, applies only to convolution and linear weights and skips biases and normalization-layer parameters, and dropout, 0.0 by default, acts just before the final linear layer of YOLO26 classification models.

from ultralytics import YOLO

# Load a pretrained YOLO26n classification model
model = YOLO("yolo26n-cls.pt")

# Train with dropout in the classification head and a stronger weight decay
results = model.train(data="mnist160", epochs=10, dropout=0.2, weight_decay=0.001)

Other Regularization Techniques#

• Data augmentation: Flips, crops, scaling, color shifts, and mosaic expand the variety of training images, so the model sees each object under many more conditions than the raw dataset contains.
• Label smoothing: Replaces one-hot classification targets with slightly softened ones, so the model stops pushing its logits toward extreme confidence.
• Implicit regularization: The gradient noise of small-batch training and the batch-to-batch noise of batch normalization both regularize without any explicit term.

Early stopping regularizes by limiting training time. It scores the model on validation data after each epoch and stops once the score has not improved for a set number of epochs, called the patience, keeping the best weights seen so far:

Regularization vs. Related Terms#

• Normalization: Rescales inputs or activations to speed up and stabilize training. Batch normalization regularizes only as a side effect.
• Hyperparameter tuning: Searches for good values of regularization settings such as λ, the dropout probability, and the patience, along with every other hyperparameter.
• Model ensembles: Reduce variance by averaging the predictions of several independently trained models. Dropout approximates such an ensemble inside one network: training samples from an exponential number of thinned subnetworks, and running the full network with every unit active at test time approximates averaging all of their predictions.