Hyperparameter Tuning

Hyperparameter tuning is the search for the training settings that give a machine learning model its best performance on held-out data. Hyperparameters are fixed before training starts, unlike the model weights that training learns from the data. Every candidate configuration has to be scored by training a model and evaluating it on validation data, so tuning is one of the most compute-intensive steps in building a model, and choosing a good search strategy matters as much as choosing the values to search.

Common Hyperparameters#

In deep learning for vision, the settings that most often need tuning fall into a few groups:

• Optimization: The learning rate and its schedule, momentum, weight decay, and the choice of optimizer, such as SGD or the Adam optimizer. The learning rate usually has the largest effect and is searched on a log scale.
• Training length and batching: The number of epochs, warmup length, and batch size.
• Data augmentation: The probability and strength of each transform, such as flips, scaling, color shifts, and mosaic.
• Loss weights: The gains that balance the terms of a composite loss function, such as box and classification losses in a detector.

Search Methods#

Practitioners navigate the space of possible configurations with a handful of strategies:

• Grid search: Trains every combination of a fixed list of values. It is simple and exhaustive, but its cost multiplies with every added hyperparameter.
• Random search: Samples configurations at random from the allowed ranges. Bergstra and Bengio showed it finds models as good as or better than grid search in a small fraction of the computation, because only a few hyperparameters matter much for any given problem and random sampling tries more distinct values of each.
• Bayesian optimization: Fits a surrogate model of how the score depends on the hyperparameters and picks each next trial where improvement looks most likely. Libraries such as Optuna use this approach by default.
• Early-termination methods: Hyperband and successive halving start many configurations on a small budget and give more epochs only to the most promising ones.
• Evolutionary algorithms: Treat each configuration as a genome and improve a population of them through selection and mutation.

Evolutionary Search#

Evolutionary search suits training runs with many interacting hyperparameters because it needs no model of the search space, only a fitness score for each trial. Each round takes one of the best configurations found so far, mutates some of its values, trains with the result, and keeps the new configuration in the pool if it scores well.

The Ultralytics tuner runs this loop for YOLO training: each iteration picks a parent from the nine highest-fitness results, mutates roughly half of up to 26 hyperparameters covering learning rate, loss gains, and augmentation, and scores fitness as mAP_50-95 for detection. The COCO-stage hyperparameters of the official YOLO26 checkpoints were found with this evolutionary search.

from ultralytics import YOLO

# Load a pretrained YOLO26n model
model = YOLO("yolo26n.pt")

# Search space: (min, max) bounds for each hyperparameter to tune
search_space = {"lr0": (1e-5, 1e-2), "degrees": (0.0, 45.0)}

# Each of the 30 iterations trains for 10 epochs with newly mutated values
model.tune(data="coco8.yaml", epochs=10, iterations=30, space=search_space, plots=False, save=False, val=False)

The run writes best_hyperparameters.yaml, a fitness-per-iteration plot, and the best iteration's weights to a tune directory. model.tune() trains with AdamW unless the optimizer is specified, because the default optimizer="auto" would override the tuned lr0.

Hyperparameter Tuning vs. Related Terms#

• Model training: Training learns the weights for one fixed set of hyperparameters through backpropagation. Tuning is the outer loop that runs training many times and compares the results.
• Fine-tuning: Despite the similar name, fine-tuning continues training a pretrained model's weights on new data. It has its own hyperparameters to tune, such as a lower learning rate and the number of frozen layers.
• Neural architecture search (NAS): Searches over the network's structure, such as layer types and widths, instead of training settings.
• Automated machine learning (AutoML): Automates the wider pipeline, including model selection and data preprocessing, with hyperparameter tuning as one component.
• Overfitting to the validation set: Hundreds of tuning trials scored on the same validation split can pick settings that suit that split by chance, so final results belong on a separate test set that tuning never touched.