Adam Optimizer

The Adam optimizer, short for Adaptive Moment Estimation, is an optimization algorithm for training neural networks that gives every parameter its own adaptive step size. Introduced by Kingma and Ba in Adam: A Method for Stochastic Optimization, it extends stochastic gradient descent (SGD) with running averages of past gradients and of their squares. Fast early progress and forgiving default settings made Adam the most common starting optimizer in deep learning, and its AdamW variant is the standard choice for training transformers and large language models.

How Adam Works#

Training minimizes a loss function by repeatedly stepping each weight against its gradient. Plain SGD scales every gradient by the same learning rate. Adam keeps two extra values per parameter and updates them at every step t from the current gradient g:

1. First moment (momentum): m = β1·m + (1 − β1)·g, an exponentially decaying average of gradients that keeps the update moving in a consistent direction.
2. Second moment: v = β2·v + (1 − β2)·g², a decaying average of squared gradients that tracks how large each parameter's gradients typically are.
3. Bias correction: m̂ = m / (1 − β1^t) and v̂ = v / (1 − β2^t), which offset the fact that both averages start at zero.
4. Update: θ = θ − lr · m̂ / (√v̂ + ε).

The paper's defaults, which PyTorch's torch.optim.Adam also uses, are lr = 0.001, β1 = 0.9, β2 = 0.999, and ε = 1e-8. Dividing by √v̂ normalizes each step by the parameter's typical gradient size, so steps stay near the learning rate whether gradients are tiny or huge, and they shrink in noisy directions where positive and negative gradients cancel inside m̂. The snippet below compares the first step of SGD and Adam on two parameters whose gradients differ in scale by 1,000x:

import torch

# Two parameters whose gradients differ in scale by 1,000x
scale = torch.tensor([1000.0, 1.0])

for name, optimizer_cls in [("SGD", torch.optim.SGD), ("Adam", torch.optim.Adam)]:
    w = torch.zeros(2, requires_grad=True)
    optimizer = optimizer_cls([w], lr=0.001)

    loss = (scale * w).sum()  # gradient is [1000, 1]
    loss.backward()
    optimizer.step()
    print(f"{name} first step: {[round(v, 4) for v in w.tolist()]}")

SGD moves the first parameter to -1.0 and the second to -0.001, a 1,000x difference that follows the gradients. Adam moves both by exactly the learning rate, -0.001, because its normalized step depends on the gradient's direction and consistency instead of its raw size.

Adam vs. AdamW#

Adam as commonly implemented applies weight decay, a form of regularization, as an L2 penalty added to the gradient, and that penalty then gets divided by √v̂ along with everything else, so weights with large gradients are barely regularized. Loshchilov and Hutter's decoupled weight decay fixes this by shrinking the weights directly at each step, separately from the adaptive update. The resulting AdamW improved Adam's generalization on image classification in the paper's experiments and is available as torch.optim.AdamW.

Adam vs. SGD#

SGD with momentum uses one global learning rate and stores one buffer per parameter, while Adam stores two, 8 bytes per parameter in 32-bit precision, which comes to 56 GB of optimizer state for a 7-billion-parameter model. Adam usually makes faster early progress, is less sensitive to the learning rate, and trains transformers far more reliably than SGD. Well-tuned SGD often reaches equal or better final accuracy on convolutional image models, and Wilson et al. found that adaptive methods often generalized worse than SGD on the image classification and language tasks they tested.

Ultralytics YOLO training reflects this trade-off: the default optimizer="auto" picks AdamW, with an initial learning rate fitted to the number of classes, for runs of up to 10,000 iterations, and switches to MuSGD, a hybrid of SGD and Muon, for longer schedules. Setting the optimizer explicitly also requires setting the learning rate, since the default lr0=0.01 suits SGD and is ten times the usual Adam rate.

from ultralytics import YOLO

# Load a pretrained YOLO26n model
model = YOLO("yolo26n.pt")

# Train with AdamW and an Adam-appropriate initial learning rate
results = model.train(data="coco8.yaml", epochs=5, optimizer="AdamW", lr0=0.001)

When YOLO26 trains with an Adam-family optimizer, the momentum argument (default 0.937) sets β1, while β2 stays fixed at 0.999.

Adam vs. Related Optimizers#

• AdaGrad: Divides each step by the root of all past squared gradients, so step sizes only ever shrink, which suits sparse features and slows long training runs.
• RMSProp: Replaces AdaGrad's running sum with an exponentially decaying average of squared gradients. Adam is essentially RMSProp plus momentum and bias correction.
• Muon: An optimizer for the 2D weight matrices of hidden layers that orthogonalizes each momentum update with Newton–Schulz iterations and keeps a single state buffer per parameter, half of Adam's optimizer state.