License YOLO
Back to Ultralytics Glossary

Double Descent

Double descent is a pattern where test error falls, peaks at the interpolation threshold, then falls again as models grow larger or train longer.

Double descent is a pattern in which a model's error on unseen data falls, rises, and then falls again as the model grows larger or trains longer. The rise peaks near the point where the model first becomes just large enough to fit every training example exactly. Past that point, adding capacity makes test error drop again, often below the best result any smaller model achieved.

The pattern extends the textbook picture of the bias-variance tradeoff, which predicts a single U-shaped curve: error falls as a model captures more structure, then climbs as it begins overfitting noise, so the best model sits somewhere in the middle. For heavily overparameterized models such as modern neural networks, that U-shape is only the first half of the curve. Mikhail Belkin and colleagues named the phenomenon in a paper published in 2019, and researchers at Harvard and OpenAI showed the same year that it appears in standard deep learning models, including ResNets and transformers.

The Interpolation Threshold#

The peak sits at the interpolation threshold, the model size at which training error first reaches zero. A model right at that size has barely enough parameters to pass through every training point, so there is essentially one way to do it. If some labels are noisy or wrong, that single solution has to bend sharply to fit them, predictions between training points become erratic, and test error spikes.

Beyond the threshold, many different parameter settings fit the training data perfectly. Gradient-based training methods such as stochastic gradient descent tend to settle on the smoothest of those solutions, in linear models exactly the one with the smallest weights, so extra capacity goes into smoothing the function between points. That implicit preference for simple solutions is why a network with far more parameters than training examples can still generalize well.

The script below reproduces the curve with random-feature regression on 40 noisy training points. Each model fits the minimum-norm least-squares solution, and the number of random features plays the role of model size:

import numpy as np

rng = np.random.default_rng(0)
n_train, dim = 40, 10
w = rng.normal(size=dim)
x_train, x_test = rng.normal(size=(n_train, dim)), rng.normal(size=(2000, dim))
y_train = x_train @ w + rng.normal(size=n_train)  # noisy training labels
y_test = x_test @ w

for n_features in [10, 20, 30, 40, 50, 100, 1000]:
    proj = rng.normal(size=(dim, n_features))
    f_train, f_test = np.maximum(x_train @ proj, 0), np.maximum(x_test @ proj, 0)

    # pinv returns the minimum-norm fit once there are more features than examples
    coef = np.linalg.pinv(f_train) @ y_train
    test_mse = np.mean((f_test @ coef - y_test) ** 2)
    print(f"{n_features:5d} features: test MSE {test_mse:8.2f}")

Test error drops from 3.37 to 2.98 between 10 and 20 features, climbs to 42.12 at 30 and about 3,960 at 40 features, where the model can exactly interpolate all 40 points, and then falls to 29.01 at 50, 2.72 at 100, and 1.33 at 1,000 features, well below the best small model.

Model-Wise, Epoch-Wise, and Sample-Wise#

Double descent shows up along three axes. Model-wise double descent, the classic curve, plots test error against model size, such as network width, at a fixed training budget. Epoch-wise double descent plots it against training time in epochs for a large model, so validation error can improve, worsen, and then improve again as training continues. Sample-wise double descent plots it against training set size: more data moves the threshold toward larger models, so adding data can temporarily hurt a model of fixed size.

Label noise makes all three more pronounced. Well-tuned regularization, such as weight decay, and strong data augmentation can shrink or remove the peak, which is one reason it is often muted in carefully configured training runs.

Two related phenomena are easy to confuse with it. In grokking, validation accuracy stays near chance long after training accuracy reaches 100% and then jumps, without first getting worse. Scaling laws describe the smooth improvement of large models with more parameters, data, and compute, the regime far to the right of the interpolation peak.

Implications for Training Vision Models#

The epoch-wise form matters most day to day: a validation metric that dips mid-training does not always mean the run has peaked. When training Ultralytics YOLO26 models in training mode, the patience argument ends a run only after 100 epochs without a gain in validation fitness by default, and the checkpoint with the best fitness is saved as best.pt, so a temporary bump neither ends the run nor replaces the best weights.

Label noise decides how sharp the model-wise peak is. In the Harvard and OpenAI experiments, ResNet18 models of increasing width trained on CIFAR-10 with clean labels showed only a plateau in test error around the interpolation threshold, and adding label noise turned that plateau into a peak; with 15% of training labels randomized, mid-sized networks tested worse than both narrower and wider ones.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning