Gradient Accumulation

Gradient accumulation is a training technique that runs several small batches, called micro-batches, through the forward pass and backpropagation, adds up their gradients, and only then updates the model weights. The update approximates one step on a batch as large as all the micro-batches combined, while the GPU only ever holds the activations of one micro-batch. It lets a memory-limited device train with the batch size a recipe calls for, without lowering image resolution or shrinking the model. The effective batch size is the micro-batch size times the number of accumulation steps times the number of devices, so micro-batches of 4 images accumulated over 8 steps on 4 GPUs behave like a batch of 128.

How Gradient Accumulation Works#

In PyTorch and most other frameworks, each call to backward() adds new gradients to those already stored on the parameters instead of replacing them. Accumulation uses that directly: run the forward and backward pass for each micro-batch, call the optimizer once the set number of micro-batches is reached, then clear the gradients and start the next window:

Most loss functions average over the samples in a batch, so each micro-batch's loss is divided by the number of steps to turn the summed gradients into an average. With that scaling, four micro-batches of 8 produce the same gradient as one batch of 32:

import torch
from torch import nn

torch.manual_seed(0)
model = nn.Linear(16, 1)
x, y = torch.randn(32, 16), torch.randn(32, 1)
loss_fn = nn.MSELoss()

# Reference: one batch of 32 samples
loss_fn(model(x), y).backward()
batch_grad = model.weight.grad.clone()
model.zero_grad()

# Four micro-batches of 8: backward() adds into .grad, and dividing each loss by 4 turns the sum into a mean
steps = 4
for x_micro, y_micro in zip(x.chunk(steps), y.chunk(steps)):
    (loss_fn(model(x_micro), y_micro) / steps).backward()

print("Matches the full-batch gradient:", torch.allclose(model.weight.grad, batch_grad, atol=1e-6))
print(f"Largest difference: {(model.weight.grad - batch_grad).abs().max():.1e}")

It prints Matches the full-batch gradient: True with a largest difference of 1.6e-07, floating-point rounding from adding the same numbers in a different order.

Where Accumulation Differs from a Large Batch#

The match holds for losses that average independent samples, and breaks in three common places:

• Batch normalization: Computes its mean and variance over each micro-batch, so statistics from 4 images are noisier than statistics from 128.
• Variable-length sequences: When a language model averages its loss over the tokens in each micro-batch, micro-batches with few tokens get the same weight as long ones.
• Gradient clipping: Belongs on the accumulated gradient, once, just before the optimizer step; clipping each micro-batch changes the result.

Hugging Face and Unsloth fixed the token-count mismatch in October 2024 for large language model (LLM) fine-tuning by dividing the summed loss by the total token count of the whole accumulation window. Accumulation also gives no speedup: the same number of samples still pass through the network one micro-batch at a time, and small micro-batches use the GPU less efficiently. In distributed training, PyTorch's DistributedDataParallel offers a no_sync() context that skips the gradient all-reduce for every micro-batch except the last, which saves communication. A larger effective batch usually calls for a higher learning rate: Facebook researchers trained ResNet-50 on ImageNet in one hour with batches of 8,192 images on 256 GPUs by scaling the learning rate linearly with batch size and warming it up gradually.

Related Techniques#

• Gradient checkpointing: Recomputes activations during backpropagation instead of storing them, saving memory within each micro-batch at the cost of extra compute.
• Mixed precision: Stores activations in 16-bit formats, which leaves room for larger micro-batches. Its loss scaler unscales the gradients once, after the last micro-batch.
• Pipeline parallelism: Splits each batch into micro-batches that flow through model stages on different GPUs and accumulates their gradients before one optimizer step.

Ultralytics YOLO26 training accumulates gradients by default: in training mode, the trainer steps the optimizer every round(nbs / batch) batches, so the default batch=16 and nominal batch size nbs=64 give four batches per update, and batch=8 gives eight. The number ramps up from one during warmup, and weight decay is scaled by batch × accumulation / nbs, so default hyperparameters set for a 64-image batch still apply on a GPU that holds only a few images at a time.