Early Stopping

Early stopping is a training technique that ends an iterative machine learning run when a monitored performance measure stops improving. It lets validation performance determine how long learning continues, reducing unnecessary computation and limiting overfitting. Intuitively, training stops when additional practice no longer improves the model's ability to handle held-out examples. Outside machine learning, "stopping early" simply means ending an activity before its planned completion.

How Early Stopping Works#

During training, an optimizer updates a model using training data. At regular intervals, the model is evaluated on separate validation data. Training loss can continue falling while validation performance deteriorates: this is a sign of overfitting, where the model increasingly fits details specific to its training examples. Early stopping limits that process by selecting a stopping point using held-out performance.

A typical implementation tracks the best validation score, saves the corresponding model weights, and counts consecutive checks without sufficient improvement. An improvement resets the counter; reaching the configured patience ends training. Restoring the best checkpoint preserves the strongest observed validation result, which can occur before the final training step.

Nelson Morgan and Hervé Bourlard introduced validation-based stopping for feedforward neural networks in their 1989 work on generalization and parameter estimation. Their experiments included speech classification and demonstrated selecting training duration by monitoring independent data.

Metrics, Patience, and Best Weights#

The stopping rule depends on several settings:

• Monitored metric: Validation loss measures prediction error and is minimized; validation accuracy measures correct predictions and is maximized.
• Patience and training duration: Patience specifies how many unsuccessful checks training tolerates. An epoch is one complete pass through the training dataset.
• Improvement threshold: Keras calls this min_delta, the minimum absolute change counted as progress. Its restore_best_weights=True setting restores the weights associated with the best monitored value.
• Validation-check frequency: PyTorch Lightning counts patience in validation checks. Checking every ten epochs with patience three permits three unsuccessful checks across thirty epochs after the best check.

Patience and improvement thresholds are hyperparameters: settings chosen outside the weight-update process. Smaller patience ends stalled runs sooner; larger patience allows longer temporary plateaus. A warm-up period delays monitoring while initial learning settles. These settings interact with the learning rate, which controls the size of optimization updates.

Applications and a Runnable Example#

Early stopping applies to both neural networks and gradient-boosted trees. In gradient boosting with scikit-learn, each stage adds a tree, and validation-based stopping controls the number of stages. The California Housing example uses validation_fraction to reserve data and n_iter_no_change to stop after validation progress stalls.

The following example uses scikit-learn's bundled diabetes dataset and prints the number of boosting stages actually fitted:

from sklearn.datasets import load_diabetes
from sklearn.ensemble import GradientBoostingRegressor

X, y = load_diabetes(return_X_y=True)

# Reserve an internal validation split and monitor progress.
model = GradientBoostingRegressor(
    n_estimators=1000,
    validation_fraction=0.2,
    n_iter_no_change=10,
    random_state=42,
)
model.fit(X, y)

print(f"Boosting stages fitted: {model.n_estimators_}/1000")

The fitted-stage count shows where the stopping criterion ended tree construction relative to the 1,000-stage ceiling.

Ultralytics YOLO applies early stopping during model training, including YOLO26, through the patience argument, which counts epochs without improved validation fitness. For detection, fitness uses mAP50–95, and the trainer saves the best checkpoint as best.pt.

Early Stopping and Related Concepts#

Early stopping is a form of regularization that controls training duration. Weight decay penalizes large weights, while dropout randomly disables units during training. These mechanisms can operate together. Stopping too soon produces underfitting, leaving useful patterns unlearned.

Checkpointing saves model states; early stopping decides when optimization ends. Cross-validation rotates held-out partitions to evaluate model choices, while separate test data supports final evaluation. Because the stopping decision selects a model using validation results, the selected checkpoint's validation score is part of model selection.

In Keras, restore_best_weights defaults to False, so enabling early stopping alone leaves the model with its final-step weights.