Gradient Descent

Gradient descent is the iterative optimization algorithm that trains most machine learning models, from linear regression to large neural networks. At each step it computes the gradient of the loss function with respect to every model weight and moves each weight a small step in the opposite direction, the direction in which the loss falls fastest. The update rule is w ← w − η∇L(w), where η is the learning rate. A common picture is a hiker descending a mountain in thick fog: unable to see the valley, the hiker feels the slope underfoot and steps downhill, again and again, until the ground levels out.

How Gradient Descent Works#

Each iteration has three parts. A forward pass computes predictions and the loss, backpropagation computes the gradient of the loss with respect to every weight, and the optimizer updates the weights. The gradient points uphill on the loss surface, following the multivariable calculus definition of the gradient, so subtracting it moves the weights downhill.

The variants differ in how much data each step uses. Batch gradient descent computes the exact gradient over the full training data set, which is accurate but slow for large datasets. Stochastic gradient descent (SGD) estimates it from a single example, and mini-batch gradient descent, the standard in deep learning, averages over a batch size of tens to thousands of examples to keep GPUs busy. The noise in mini-batch gradients also helps the weights move past saddle points and sharp minima. One full pass through the training set is an epoch.

Choosing the Learning Rate#

The learning rate sets the step size, and it is usually the first hyperparameter to tune in gradient descent. Too small a rate wastes compute creeping toward the minimum, while too large a rate overshoots it; on a quadratic loss, any rate above 2 divided by the loss curvature makes every step land farther from the minimum than the last. The snippet below minimizes (w − 2)², whose curvature is 2, with three learning rates in PyTorch:

import torch

# Minimize loss(w) = (w - 2)^2, whose minimum is at w = 2, with three learning rates
for lr in [0.1, 0.5, 1.1]:
    w = torch.tensor(5.0, requires_grad=True)
    for step in range(20):
        loss = (w - 2) ** 2
        loss.backward()  # w.grad now holds the slope 2 * (w - 2)
        with torch.no_grad():
            w -= lr * w.grad  # step against the gradient
        w.grad.zero_()
    print(f"Learning rate {lr}: w = {w.item():.4f} after 20 steps")

With a rate of 0.1 the weight is still at 2.0346 after 20 steps, a rate of 0.5 lands exactly on 2.0000 because it equals one over the curvature, and a rate of 1.1 overshoots by a growing margin each step until w reaches 117.0129. Real training schedules start with a short warmup and then decay the rate, often with a cosine curve, so early steps stay stable and late steps settle precisely.

Momentum, Adam, and Modern Optimizers#

Plain gradient descent zigzags across narrow valleys of the loss surface, so most training uses improved update rules. Momentum accumulates a running average of past gradients and moves along it, which damps oscillation and speeds progress along consistent directions. The Adam optimizer also tracks each parameter's squared gradients to give every weight its own adaptive step size, and AdamW decouples weight decay from that update. Muon orthogonalizes the momentum update of each 2D weight matrix with Newton-Schulz iterations so that no single direction dominates the step. Ultralytics YOLO26 training with optimizer="auto" selects AdamW, with a learning rate fitted to the number of classes, for runs of up to 10,000 iterations, and MuSGD, which adds Muon-style orthogonalized updates to SGD for convolution and linear weights, for longer runs.

Real-World Applications#

Gradient descent is the training engine behind nearly every deep learning system. In autonomous vehicles, it tunes object detection networks to shrink the gap between predicted and labeled boxes for pedestrians, signs, and vehicles across millions of frames. In medical image analysis, it fits convolutional neural networks that separate tumors from healthy tissue in MRI and CT scans. Large language models are trained the same way, typically with AdamW over trillions of tokens.

Gradient Descent vs. Related Concepts#

• Backpropagation: Backpropagation computes the gradients using the chain rule, and gradient descent uses them to update the weights. The two run back to back in every training step and are often mentioned together.
• Loss Function: The loss function scores how wrong the model's predictions are, and gradient descent is the procedure that lowers that score.
• Exploding Gradient: When gradients grow uncontrollably in deep networks, gradient descent takes destructive steps, which gradient clipping prevents by capping the gradient norm before each update.
• Evolutionary Algorithms: These gradient-free methods search by sampling and selecting candidates without computing gradients, which suits objectives that are not differentiable, such as hyperparameter choices.