Optimization Algorithm

An optimization algorithm, or optimizer, is the rule a training process uses to change a model's parameters so that its loss function decreases. In deep learning, the optimizer receives the gradient of the loss with respect to every model weight from backpropagation and decides how far, and in which direction, to move each one. Together with its learning rate, the optimizer determines how fast training converges, how stable it is, how much memory it needs, and often how well the final model generalizes.

How Optimizers Work#

Nearly every deep learning optimizer is a variant of gradient descent: compute the gradient g of the loss on a mini-batch, then move the weights a small step against it, w ← w − η·g. Optimizers differ in what they do with g before stepping. Momentum methods average gradients over time to build speed along directions that stay consistent. Adaptive methods divide each parameter's step by a running estimate of its gradient size. Muon orthogonalizes the update for each weight matrix as a whole. Second-order methods also use curvature, which makes each step better informed and far more expensive to compute. Methods that need no gradient at all, such as evolutionary algorithms and random search, optimize by trying candidates and keeping the best:

The update rule matters most when the loss surface is badly conditioned, curving steeply in some directions and gently in others. Running three optimizers from the same starting point in a narrow valley shows the difference:

import torch

# A long, narrow valley: f(w) = w1^2 + 50 * w2^2, with its minimum at (0, 0)
scale = torch.tensor([1.0, 50.0])
optimizers = {
    "SGD": lambda p: torch.optim.SGD(p, lr=0.015),
    "SGD + momentum": lambda p: torch.optim.SGD(p, lr=0.015, momentum=0.9),
    "Adam": lambda p: torch.optim.Adam(p, lr=0.1),
}

for name, make_optimizer in optimizers.items():
    w = torch.tensor([5.0, 1.0], requires_grad=True)
    optimizer = make_optimizer([w])
    # Same loss and starting point for every optimizer; only the update rule differs
    for step in range(100):
        loss = (scale * w**2).sum()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f"{name:15s} f(w) after 100 steps: {(scale * w**2).sum().item():.6f}")

Plain SGD stops at 0.056531 because the steep w2 direction caps its learning rate: any rate above 0.02 makes w2 overshoot by more on every step, and at 0.021 the loss climbs to about 9.5 billion within 100 steps. Under that cap, progress along the gentle w1 direction is slow. Momentum reaches 0.000216 by building speed along w1, and Adam reaches 0.001953 by giving each coordinate its own step size.

Common Optimization Algorithms#

• Stochastic gradient descent (SGD): Steps against the mini-batch gradient. With momentum 0.9, it trained most landmark convolutional networks and stores one extra value per parameter.
• AdaGrad and RMSProp: Scale each parameter's step by the history of its squared gradients. AdaGrad divides by the root of their running sum, which only grows, so its steps shrink throughout training. RMSProp uses an exponentially decaying average instead, which keeps step sizes responsive.
• Adam: Combines momentum with RMSProp-style scaling and corrects both running averages for their zero start. Its defaults, a learning rate of 0.001 with β1 = 0.9 and β2 = 0.999, work across many problems.
• AdamW: Adam with decoupled weight decay, which shrinks weights directly instead of through the adaptive gradient scaling. It is the standard optimizer for transformers and large language models (LLMs).
• Muon: Orthogonalizes the momentum update of each 2D weight matrix with Newton–Schulz iterations and keeps a single state buffer per parameter. Moonshot AI scaled it to LLM pretraining.
• L-BFGS: A quasi-Newton method that builds a curvature estimate from recent gradients. It converges in few steps on smooth, full-batch problems, and noisy mini-batch gradients make it a poor fit for large networks.

PyTorch's optimizer package implements all of these, including torch.optim.Muon, along with learning rate schedulers that work with any of them.

Optimizer vs. Related Terms#

• Loss function: The objective the optimizer minimizes, such as cross-entropy for classification or mean squared error for regression.
• Learning rate scheduler: Adjusts the step size η over training, with warmup, step decay, or cosine annealing, on top of any optimizer.
• Hyperparameter tuning: An outer search over settings such as the optimizer, learning rate, and batch size, where each candidate costs a full or partial training run. Gradient-free methods such as Bayesian optimization and evolutionary search do this job, because a validation score cannot be differentiated with respect to choices such as the optimizer.

Choosing an Optimizer#

The optimizer and learning rate are chosen together, because each optimizer works in its own range: SGD with momentum typically starts between 0.01 and 0.1, and Adam-family optimizers around 0.001. Ultralytics YOLO training makes the choice automatically: the default optimizer="auto" ignores lr0 and picks AdamW with a learning rate fitted to the number of classes for runs of up to 10,000 iterations, or MuSGD, the SGD and Muon hybrid used to train the released YOLO26 checkpoints, for longer schedules, while naming an optimizer makes lr0 take effect:

from ultralytics import YOLO

# Load a pretrained YOLO26n model
model = YOLO("yolo26n.pt")

# Choose SGD explicitly; lr0 only applies when the optimizer is not "auto"
results = model.train(data="coco8.yaml", epochs=5, optimizer="SGD", lr0=0.01)

With optimizer="SGD", the trainer also enables Nesterov momentum, using the momentum argument, 0.937 by default, as the coefficient.