Bias-Variance Tradeoff

The bias-variance tradeoff describes how a model's expected error on new data splits into two parts that respond to model complexity in opposite directions. Bias is the error from assumptions too simple to capture the real pattern, which causes underfitting. Variance is the error from sensitivity to the particular training data, which causes overfitting. Making a model more flexible usually lowers bias and raises variance, so the lowest total error sits at an intermediate complexity. Stuart Geman, Elie Bienenstock, and René Doursat brought the idea into machine learning in their 1992 paper "Neural Networks and the Bias/Variance Dilemma."

The Error Decomposition#

The decomposition considers the same kind of model trained many times, each time on a fresh training set drawn from the same source. For squared error, the expected error at an input splits exactly into three terms. Bias squared is the squared gap between the average prediction over all those training sets and the true value; a linear regression fit to a curved relationship keeps a high bias no matter how much data it sees. Variance measures how much the predictions scatter around their own average from one training set to the next, and it is high for deep decision trees and large unregularized neural networks. The third term, irreducible noise, is the randomness in the labels themselves, which no model can remove.

Repeating the fit measures both terms directly. This NumPy example fits polynomials of degree 1, 3, and 12 to 1,000 training sets of 20 noisy points from a sine wave:

import numpy as np

rng = np.random.default_rng(0)
x = np.linspace(0, 1, 20)  # 20 training inputs, evenly spaced
x_test = np.linspace(0, 1, 200)
true_curve = np.sin(2 * np.pi * x_test)

for degree in [1, 3, 12]:
    # Fit the same polynomial degree to 1,000 training sets that differ only in their label noise
    predictions = []
    for _ in range(1000):
        y = np.sin(2 * np.pi * x) + rng.normal(0, 0.3, x.size)
        predictions.append(np.polyval(np.polyfit(x, y, degree), x_test))
    predictions = np.array(predictions)

    # Bias^2: squared gap between the average fit and the truth; variance: spread of fits around their average
    bias_sq = np.mean((predictions.mean(axis=0) - true_curve) ** 2)
    variance = np.mean(predictions.var(axis=0))
    print(f"Degree {degree:2d}: bias^2 = {bias_sq:.3f}, variance = {variance:.3f}")

A straight line has a bias squared of 0.205 and a variance of only 0.008. A cubic matches the sine almost exactly on average, cutting bias squared to 0.005 while variance doubles to 0.016. A degree-12 polynomial has no measurable bias but a variance of 0.074, because it bends to follow the noise in each sample. The cubic has the lowest total error, and every model also pays the irreducible noise variance of 0.09.

Diagnosing Bias and Variance#

Training and validation errors reveal which term dominates. High error on both points to bias. Low training error with much higher validation error points to variance. Learning curves, which plot both errors against the amount of training data, separate the two cases: a high-variance model's gap narrows as data grows, while a high-bias model's two curves meet early at a high error and stay there. In Ultralytics YOLO training, the fraction argument trains on part of the training split, such as fraction=0.25 for the first quarter of its images, while validation still uses the full validation split, so a few YOLO26 runs at increasing fractions trace a learning curve for a custom dataset.

Controlling the Tradeoff#

Variance falls with more data, data augmentation, regularization such as weight decay and dropout, early stopping, and averaging many models, as bagging and random forests do. Bias falls with a larger model, richer feature engineering, weaker regularization, longer training, and boosting, which adds models that correct the errors of the ensemble so far. Hyperparameter tuning and cross-validation choose the setting where the two terms balance on held-out data.

The classic U-shaped curve of test error against complexity holds for a fixed family of models, such as polynomials of rising degree. Heavily overparameterized neural networks can go past it: test error rises as a model approaches the size that fits every training point exactly and then falls again as it grows further, a pattern called double descent. Bias and variance still add up to the error there, but variance no longer grows steadily with parameter count.

Statistical Bias and Other Kinds of Bias#

The bias in this tradeoff is a property of a learning algorithm on a task. Bias in AI refers to systematic unfairness toward groups of people, and dataset bias to training data that misrepresents the deployment setting, as when nighttime traffic footage is scarce. A model can have low statistical bias and still be unfair, because it can fit a skewed dataset precisely, skew included.