Gradient Accumulation

Gradient accumulation is a training technique that adds gradients from several small batches, called microbatches, before performing a single model-weight update. It approximates training with a larger batch size when available GPU memory cannot hold the full batch at once. This makes memory-intensive gradient descent practical without changing the model architecture or reducing input resolution.

How Gradient Accumulation Works#

During normal training, each batch follows three main steps: compute the loss, use backpropagation to calculate gradients, and let the optimizer update the model weights. The gradients are then cleared before the next batch.

Gradient accumulation delays the optimizer update. Each microbatch performs a forward and backward pass, but its gradients remain in memory and are added to those from subsequent microbatches. This behavior follows naturally from the PyTorch backward operation, which accumulates values in each parameter’s gradient field.

After the configured number of accumulation steps:

1. The optimizer updates the model weights.
2. The gradients are reset using an operation such as Optimizer.zero_grad.
3. A new accumulation window begins.

The effective batch size is:

microbatch size x accumulation steps x number of training devices

For example, a microbatch of 4 images accumulated over 8 steps behaves approximately like a batch of 32 images on one GPU. With four GPUs, the global effective batch size becomes 128.

Effects on Memory, Speed, and Learning#

Gradient accumulation reduces the peak memory required for activations because only one microbatch passes through the network at a time. It does not substantially reduce memory used by model parameters, gradients, or optimizer state.

It also does not usually speed up training. Processing eight microbatches still requires eight forward and backward passes, and smaller batches may use the GPU less efficiently. The main benefit is fitting a desired effective batch into limited memory. A secondary benefit can occur in distributed training, where implementations may avoid synchronizing gradients until the end of an accumulation window using features such as DistributedDataParallel no_sync.

A manual implementation commonly divides each microbatch loss by the number of accumulation steps so the resulting gradient represents an average rather than a sum. High-level trainers may handle normalization automatically, so applying additional scaling can produce incorrect updates.

Accumulation can approximate a true large batch, but the results are not always identical. Batch normalization statistics are calculated from individual microbatches, while stochastic operations and floating-point ordering can also introduce differences. The learning rate may therefore require validation rather than automatic scaling.

Related Techniques and Key Differences#

Gradient accumulation is often confused with other memory or stability techniques:

• Gradient checkpointing reduces activation memory by recomputing selected forward-pass operations during backpropagation. The PyTorch activation checkpointing documentation describes this compute-for-memory tradeoff. Accumulation instead divides a large batch into smaller microbatches.
• Mixed precision stores or computes selected operations with lower-precision data types. It can reduce memory and improve throughput, while accumulation changes how often the optimizer updates. The techniques can be combined by following the correct automatic mixed precision accumulation workflow.
• Gradient clipping limits gradient magnitude to reduce instability from unusually large updates. It does not create a larger effective batch. When combined with accumulation, clipping should normally occur after the complete gradient has been accumulated, using an operation such as clip_grad_norm.

Real-World Applications#

In high-resolution medical image segmentation, a 12 GB GPU may fit only two large scans at once. Accumulating gradients across eight microbatches produces an effective batch of 16 while preserving the resolution needed to identify small anatomical structures. The tradeoff is a longer interval between optimizer updates.

For aerial object detection, an engineering team might train across four GPUs with six images per device and four accumulation steps. The effective global batch is 96 images. This can stabilize updates for scenes containing many small vehicles or buildings while keeping each device within its memory limit. Teams can manage cloud experiments, datasets, and training runs through Ultralytics Platform cloud training.

Gradient Accumulation with Ultralytics YOLO#

The Ultralytics model training workflow derives accumulation behavior from the physical batch and nominal nbs settings. With batch=4 and nbs=64, the trainer targets a nominal batch of 64 by deferring optimizer updates across multiple microbatches.

from ultralytics import YOLO

# Load a pretrained YOLO26 detection model
model = YOLO("yolo26n.pt")

# Use a small physical batch and a larger nominal batch
results = model.train(
    data="coco8.yaml",
    epochs=10,
    batch=4,
    nbs=64,
)

The high-level trainer manages backpropagation, accumulation, optimizer steps, and gradient clearing. Its behavior is documented in the Ultralytics BaseTrainer reference. In practice, choose the largest microbatch that fits reliably, calculate the intended effective batch, and compare validation results when changing accumulation steps, learning rate, or device count.