Bayesian Optimization

Bayesian optimization is a method for finding high-performing settings of an expensive-to-evaluate function by learning from previous evaluations. It builds a probabilistic model of how inputs affect results, then uses that model to choose the next experiment. In machine learning, this makes it useful for finding strong training configurations within a limited computational budget.

Its foundations include Harold Kushner's 1964 sequential search method and Jonas Mockus's 1975 Bayesian extremum-seeking formulation. Today, it is a central technique in hyperparameter tuning and automated machine learning.

How Bayesian Optimization Works#

The objective function measures the quality of a candidate configuration. For model training, an evaluation involves training with selected hyperparameters and measuring performance on validation data. Tunable inputs include learning rate, batch size, and regularization strength.

A surrogate model is a computationally cheaper approximation of the objective. A common choice is a Gaussian process: a probability distribution over possible functions that predicts both an outcome and its uncertainty. Bayesian updating combines prior assumptions with observed evaluations to produce a posterior distribution. An acquisition function uses this distribution to select another candidate; its measured result feeds back into the surrogate until the evaluation budget is exhausted.

This is a machine-learning-based optimization approach: the surrogate learns from configuration–result pairs. Its uncertainty quantification helps determine which additional measurements are valuable.

Acquisition Functions and Exploration#

Candidate selection balances exploitation, which favors settings predicted to perform well, with exploration, which investigates uncertain regions. Common acquisition functions express this balance differently:

• Expected Improvement (EI): Scores the expected amount by which a candidate improves on a reference result.
• Probability of Improvement (PI): Scores the probability of exceeding a reference result by a specified margin.
• Upper Confidence Bound (UCB): For maximization, combines the predicted mean with an uncertainty bonus, giving uncertain candidates additional priority.

These acquisition strategies are inexpensive to evaluate compared with training a model or running a physical experiment. Their optimization directs the next expensive evaluation.

Applications in Machine Learning and Computer Vision#

Bayesian optimization is especially useful when individual evaluations consume substantial time and the search contains a manageable number of influential variables. Established applications include engineering design, materials experiments, and reinforcement learning controller tuning.

In computer vision, it selects training settings for convolutional neural networks. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams demonstrated this application in their 2012 practical Bayesian optimization algorithm, alongside tuning structured support vector machines and topic models.

Ultralytics YOLO exposes Bayesian search through its Ray Tune integration: setting use_ray=True and search_alg="bayesopt" selects Bayesian optimization for training hyperparameters. For YOLO26 object detection, the search maximizes the task's default mean average precision metric.

Related Methods and Optuna#

Grid search evaluates a predefined collection of combinations, while random search samples configurations independently. Bayesian optimization uses previous outcomes to guide subsequent choices. Gradient descent updates parameters using derivatives; Bayesian optimization handles expensive black-box objectives through evaluations and a surrogate.

Optuna's sampling algorithms include Gaussian-process Bayesian optimization, random sampling, and the Tree-structured Parzen Estimator (TPE). TPE models parameter distributions associated with better and worse outcomes, then favors candidates with a larger ratio between those densities. Optuna is the optimization framework; the selected sampler determines its search strategy.

A Python Example#

Install bayesian-optimization with pip install bayesian-optimization. This uses the objective and bounds from the package's introductory optimization example:

from bayes_opt import BayesianOptimization


def objective(x, y):
    """Return the objective value to maximize."""
    return -(x**2) - (y - 1) ** 2 + 1


optimizer = BayesianOptimization(
    f=objective,
    pbounds={"x": (2, 4), "y": (-3, 3)},
    random_state=1,
    verbose=0,
)
optimizer.maximize(init_points=2, n_iter=15)
best = optimizer.max
print(f"Best objective: {best['target']:.4f}")

The output reports the best measured objective after two random evaluations and fifteen acquisition-guided evaluations. Within these bounds, the exact maximum is -3 at x=2, y=1.