Double Descent
Double descent is a pattern where test error falls, peaks at the interpolation threshold, then falls again as models grow larger or train longer.
Double descent is a pattern in which a model's error on unseen data falls, rises, and then falls again as the model grows larger or trains longer. The rise peaks near the point where the model first becomes just large enough to fit every training example exactly. Past that point, adding capacity makes test error drop again, often below the best result any smaller model achieved.
The pattern extends the textbook picture of the bias-variance tradeoff, which predicts a single U-shaped curve: error falls as a model captures more structure, then climbs as it begins overfitting noise, so the best model sits somewhere in the middle. For heavily overparameterized models such as modern neural networks, that U-shape is only the first half of the curve. Mikhail Belkin and colleagues named the phenomenon in a paper published in 2019, and researchers at Harvard and OpenAI showed the same year that it appears in standard deep learning models, including ResNets and transformers.
The Interpolation Threshold#
The peak sits at the interpolation threshold, the model size at which training error first reaches zero. A model right at that size has barely enough parameters to pass through every training point, so there is essentially one way to do it. If some labels are noisy or wrong, that single solution has to bend sharply to fit them, predictions between training points become erratic, and test error spikes.
Beyond the threshold, many different parameter settings fit the training data perfectly. Gradient-based training methods such as stochastic gradient descent tend to settle on the smoothest of those solutions, in linear models exactly the one with the smallest weights, so extra capacity goes into smoothing the function between points. That implicit preference for simple solutions is why a network with far more parameters than training examples can still generalize well.
The script below reproduces the curve with random-feature regression on 40 noisy training points. Each model fits the minimum-norm least-squares solution, and the number of random features plays the role of model size:
import numpy as np
rng = np.random.default_rng(0)
n_train, dim = 40, 10
w = rng.normal(size=dim)
x_train, x_test = rng.normal(size=(n_train, dim)), rng.normal(size=(2000, dim))
y_train = x_train @ w + rng.normal(size=n_train) # noisy training labels
y_test = x_test @ w
for n_features in [10, 20, 30, 40, 50, 100, 1000]:
proj = rng.normal(size=(dim, n_features))
f_train, f_test = np.maximum(x_train @ proj, 0), np.maximum(x_test @ proj, 0)
# pinv returns the minimum-norm fit once there are more features than examples
coef = np.linalg.pinv(f_train) @ y_train
test_mse = np.mean((f_test @ coef - y_test) ** 2)
print(f"{n_features:5d} features: test MSE {test_mse:8.2f}")Test error drops from 3.37 to 2.98 between 10 and 20 features, climbs to 42.12 at 30 and about 3,960 at 40 features, where the model can exactly interpolate all 40 points, and then falls to 29.01 at 50, 2.72 at 100, and 1.33 at 1,000 features, well below the best small model.
Model-Wise, Epoch-Wise, and Sample-Wise#
Double descent shows up along three axes. Model-wise double descent, the classic curve, plots test error against model size, such as network width, at a fixed training budget. Epoch-wise double descent plots it against training time in epochs for a large model, so validation error can improve, worsen, and then improve again as training continues. Sample-wise double descent plots it against training set size: more data moves the threshold toward larger models, so adding data can temporarily hurt a model of fixed size.
Label noise makes all three more pronounced. Well-tuned regularization, such as weight decay, and strong data augmentation can shrink or remove the peak, which is one reason it is often muted in carefully configured training runs.
Two related phenomena are easy to confuse with it. In grokking, validation accuracy stays near chance long after training accuracy reaches 100% and then jumps, without first getting worse. Scaling laws describe the smooth improvement of large models with more parameters, data, and compute, the regime far to the right of the interpolation peak.
Implications for Training Vision Models#
The epoch-wise form matters most day to day: a validation metric that dips mid-training does not always mean the run has peaked. When training Ultralytics YOLO26 models in training mode, the patience argument ends a run only after 100 epochs without a gain in validation fitness by default, and the checkpoint with the best fitness is saved as best.pt, so a temporary bump neither ends the run nor replaces the best weights.
Label noise decides how sharp the model-wise peak is. In the Harvard and OpenAI experiments, ResNet18 models of increasing width trained on CIFAR-10 with clean labels showed only a plateau in test error around the interpolation threshold, and adding label noise turned that plateau into a peak; with 15% of training labels randomized, mid-sized networks tested worse than both narrower and wider ones.










