Learning Rate

The learning rate is the hyperparameter that sets how far a model's weights move on each training step. Gradient descent computes the gradient of the loss function, which points in the direction that increases the loss fastest, and updates each weight as w ← w − η × gradient, where η (eta) is the learning rate. Step size is a synonym, and code usually calls it lr. A well-chosen rate lets a neural network converge quickly to a good solution, and a badly chosen one makes training diverge or crawl, which is why Leslie Smith's cyclical learning rates paper calls it the most important hyperparameter to tune for deep networks.

Too High, Too Low, and Just Right#

The learning rate trades speed against stability:

• Too high: Each step overshoots the minimum. The loss oscillates, jumps, or grows until it becomes NaN, and training diverges.
• Too low: Each step barely moves the weights. The loss falls slowly, training needs far more epochs to converge, and a run with a fixed budget ends underfit.
• Well chosen: The loss falls steadily and settles near a minimum.

The PyTorch script below minimizes the loss w² from the same starting point with four learning rates:

import torch

# Minimize loss = w**2 (lowest at w = 0) from the same start with four learning rates
for lr in (0.01, 0.1, 0.9, 1.1):
    w = torch.tensor(5.0, requires_grad=True)
    optimizer = torch.optim.SGD([w], lr=lr)
    path = []
    for step in range(20):
        optimizer.zero_grad()
        loss = w**2
        loss.backward()
        optimizer.step()  # w = w - lr * gradient
        path.append(w.item())
    print(f"lr={lr}: first steps {[round(v, 2) for v in path[:3]]}, after 20 steps w={path[-1]:.3f}")

At 0.01 the weight only creeps from 5 to 3.338 in 20 steps, and at 0.1 it reaches 0.058. At 0.9 it flips sign on every step (−4.0, 3.2, −2.56) yet still converges to 0.058, while at 1.1 each step overshoots further and the weight grows to 191.688. Gradient descent on a quadratic stays stable only while the learning rate is below 2 divided by the loss's curvature, its second derivative. The curvature of w² is 2, so the limit here is exactly 1.0, and a rate of 0.5 lands on the minimum in one step. In a real network the sharpest direction of the loss sets the same kind of limit, which is why a rate that works for one model or batch size can diverge for another.

Learning Rate Schedules#

Training rarely keeps one rate from start to finish. A schedule changes it over time:

• Warmup: Starts from a small rate and ramps up over the first few epochs or iterations, which keeps the large, erratic gradients of the first iterations from destabilizing training.
• Step decay: Divides the rate by a fixed factor, often 10, at set epochs or when the validation loss plateaus.
• Linear and cosine decay: Lowers the rate smoothly toward a small final value. Cosine annealing follows half a cosine wave, and its warm-restart variant jumps back to the high rate at the start of each new cycle.
• Cyclical and one-cycle policies: Move the rate between a lower and an upper bound during training.

Large early steps move the weights quickly toward a good region, and small late steps let them settle into a minimum instead of bouncing around it. YOLO26 training in Ultralytics starts at lr0, warms up over warmup_epochs (3 by default), and decays linearly to lr0 * lrf, or along a cosine with cos_lr=True. With the default optimizer="auto" the trainer chooses both the optimizer and its initial rate itself, MuSGD at 0.01 for runs longer than 10,000 iterations and otherwise AdamW at a rate fitted to the number of classes, so lr0 applies only when the optimizer is set explicitly:

from ultralytics import YOLO

# Load a pretrained YOLO26 nano model
model = YOLO("yolo26n.pt")

# An explicit optimizer makes lr0 apply; the rate decays to lr0 * lrf along a cosine
results = model.train(data="coco8.yaml", epochs=10, optimizer="SGD", lr0=0.01, lrf=0.01, cos_lr=True)

Choosing a Starting Value#

Common starting points are 0.01 to 0.1 for stochastic gradient descent (SGD) with momentum and 0.001 for the Adam optimizer, the default proposed in the Adam paper. Fine-tuning a pretrained model uses a smaller rate than training from scratch, because large steps would overwrite the features that make transfer learning work.

The learning rate range test from the same cyclical learning rates paper, packaged in fastai as its learning rate finder, finds a good value in one short run. It raises the rate after every mini-batch while recording the loss, and the useful range lies where the loss falls fastest, before it starts to explode.

Learning Rate vs. Related Terms#

• Optimization algorithm: The update rule that uses the learning rate. Adaptive optimizers such as Adam scale each weight's step by its own gradient history but still multiply every step by one global learning rate.
• Momentum: A separate hyperparameter that adds a fraction of the previous update to the current one, smoothing the path and raising the effective step size along consistent directions.
• Weight decay: A regularization term that shrinks the weights themselves toward zero on every step, while learning rate decay shrinks the step size over time.
• Batch size: The number of examples averaged into each gradient. Larger batches give less noisy gradients and tolerate larger rates, and the linear scaling rule multiplies the learning rate by k when the batch size grows by k.