Batch Size

Batch size is the number of training examples a model processes together before it updates its model weights. Instead of feeding the whole training dataset through a neural network at once, training splits it into batches. For each batch, the model makes predictions, the loss function scores them, backpropagation computes gradients, and the optimizer takes one step. Batch size is a hyperparameter that governs three things at once: how much GPU memory training needs, how fast each epoch runs, and how noisy each gradient estimate is.

How Batch Size Affects Training#

Each batch gives the optimization algorithm an estimate of the gradient over the full dataset, and the estimate gets more precise as the batch grows. The PyTorch snippet below measures that precision directly by averaging per-sample gradients over many random batches of each size:

import torch

torch.manual_seed(0)

# Per-sample gradients of a squared-error loss for 10,000 samples
x = torch.randn(10_000)
y = 2 * x + torch.randn(10_000)
w = torch.tensor(0.0)
per_sample_grads = 2 * (w * x - y) * x

# Estimate the gradient from 1,000 random batches at each batch size
for batch_size in (8, 64, 512):
    idx = torch.randint(0, len(x), (1000, batch_size))
    batch_grads = per_sample_grads[idx].mean(dim=1)
    print(f"batch {batch_size:3d}: gradient std {batch_grads.std():.3f}")

The spread of the gradient estimate falls from 2.248 at batch size 8 to 0.771 at 64 and 0.267 at 512. Each 8x increase in batch size cuts the noise by about 2.9x, close to the square root of 8, which is how averaging independent samples behaves.

That noise shapes the trade-off between small and large batches:

• Small batches (for example, 8 to 32): Noisy updates and many steps per epoch. The noise acts as a mild regularizer that often helps stochastic gradient descent (SGD) generalize, and small batches fit on modest hardware, but each epoch takes longer because the GPU sits partly idle.
• Large batches (for example, 256 and up): Smooth gradients, fewer steps per epoch, and high hardware utilization. Very large batches tend to converge to sharp minima that generalize worse, as Keskar et al. found, and because they take fewer steps per epoch, they need a proportionally larger learning rate with a warmup period. Using that linear scaling rule, Goyal et al. trained ResNet-50 on ImageNet with a batch size of 8,192 across 256 GPUs in one hour while matching the accuracy of small-batch training.

Memory Limits and Gradient Accumulation#

GPU memory must hold the model's weights, its optimizer state, and the intermediate activations that backpropagation needs, and activation memory grows linearly with batch size and image resolution. A batch that does not fit stops training with an out-of-memory (OOM) error. Mixed precision training roughly halves activation memory, and gradient accumulation reaches a larger effective batch by summing gradients over several small batches before each weight update.

Ultralytics YOLO training builds this in: it accumulates gradients until it has seen about 64 images, the nominal batch size nbs, so after warmup the default batch=16 makes one weight update every four batches. Setting batch=-1 runs AutoBatch, which picks the largest batch that fits in about 60% of free CUDA memory and falls back to a batch size of 16 on CPU and Apple MPS devices.

from ultralytics import YOLO

# Load a pretrained YOLO26n model
model = YOLO("yolo26n.pt")

# batch=-1 lets AutoBatch choose the batch size from available GPU memory
# Use an integer (batch=16) for a fixed size or a float (batch=0.70) for a memory fraction
results = model.train(data="coco8.yaml", epochs=5, batch=-1)

Before training starts, AutoBatch runs the YOLO26 model at batch sizes from 1 to 16 (up to 64 on GPUs with 16 GB or more), fits a straight line to the measured memory use, and solves it for the batch size that hits the target fraction.

Batch Size vs. Related Terms#

• Epoch and iteration: An epoch is one pass over the full dataset, and an iteration is one weight update. Iterations per epoch equal the dataset size divided by the batch size, rounded up, so 1,000 images at batch size 100 take 10 iterations.
• Mini-batch: A synonym for batch in most deep learning writing. Strictly, batch gradient descent uses the entire dataset per update, stochastic gradient descent uses one example, and mini-batch gradient descent sits in between, which is how nearly all modern networks train.
• Batch normalization: A layer that normalizes activations with the mean and variance of the current batch, so its statistics get unreliable at very small batch sizes. Group Normalization avoids the dependency and reported 10.6% lower error than batch normalization for ResNet-50 on ImageNet at a batch size of 2.
• Inference batch size: At deployment time, batching trades inference latency for throughput. Grouping frames from several cameras raises the number of images processed per second, while each frame waits for its batch to fill.