Distributed Training

Distributed training spreads the work of training one model across multiple processors, usually GPUs or TPUs in one machine or across a cloud or on-premise cluster, so that training finishes sooner or a model too large for one device's memory can be trained at all. The devices exchange gradients, activations, or parameter shards over fast interconnects using communication libraries such as NVIDIA NCCL. It is how modern foundation models and large language models are trained, and it also shortens everyday object detection training on multi-GPU workstations.

How Distributed Training Works#

Data parallelism is the most common strategy. Every GPU holds a full replica of the model, processes its own slice of each batch of training data, and joins an all-reduce operation that averages the gradients so every replica applies the same gradient descent update. PyTorch's DistributedDataParallel (DDP) overlaps that communication with the backward pass. The global batch size is the per-GPU batch times the number of GPUs, so the learning rate usually has to scale with it: Goyal et al. trained ResNet-50 on ImageNet in one hour on 256 GPUs with a batch of 8,192 images, using a linear scaling rule and a warmup phase, at about 90% scaling efficiency from 8 to 256 GPUs.

Sharded data parallelism cuts the memory cost of those replicas. ZeRO and PyTorch's Fully Sharded Data Parallel (FSDP) split the parameters, gradients, and optimizer states across GPUs and gather each layer's full weights only while that layer is computing, so the memory each GPU needs falls as GPUs are added.

Model parallelism splits the neural network itself. Tensor parallelism divides individual matrix multiplications across GPUs, pipeline parallelism assigns consecutive layers to different GPUs and streams micro-batches through them, context parallelism splits long sequences, and expert parallelism places the experts of a mixture of experts model on different GPUs. Frameworks such as Megatron-LM combine these with data parallelism to train models with hundreds of billions of parameters.

Multi-GPU Training in Practice#

Communication is the main overhead: all-reduce traffic grows with model size, so GPUs linked by NVLink within a node scale better than nodes connected over slower networks. Batch normalization statistics are computed per GPU unless layers are converted to synchronized batch normalization, which matters when each GPU's slice of the batch is small. Ultralytics YOLO26 runs multi-GPU training through DDP: passing several device IDs launches one process per GPU with torch.distributed.run, using the NCCL backend on CUDA, and splits the batch value evenly across the GPUs.

from ultralytics import YOLO

# Load a pretrained YOLO26 model
model = YOLO("yolo26n.pt")

# Train on GPUs 0 and 1 with DDP; batch=32 gives each GPU 16 images per step
results = model.train(data="coco8.yaml", epochs=100, imgsz=640, batch=32, device=[0, 1])

Real-World Applications#

• Large Language Models: Meta trained Llama 3 405B on up to 16K H100 GPUs, combining tensor, pipeline, context, and data parallelism in what its authors call 4D parallelism.
• Autonomous Vehicles: Perception teams train detection and segmentation models on large archives of camera, lidar, and radar recordings from vehicle fleets, which is only practical on large GPU clusters.
• Medical Image Analysis: High-resolution 3D CT and MRI volumes strain the memory of a single GPU, and spreading batches or model shards across GPUs makes larger volumes and models trainable.

Distributed Training vs. Related Concepts#

• Federated Learning: Federated learning also trains across many devices, but the data stays on phones, hospitals, or other clients for privacy, and only model updates travel to a central server. Distributed training gathers data in one cluster to maximize speed.
• Split Learning: Split learning divides a model between a client and a server so raw data never leaves the client, trading extra communication for privacy.
• Gradient Accumulation: Gradient accumulation sums gradients over several small batches before one update, matching a large batch's math on one GPU at the cost of time, while data parallelism computes those batches simultaneously on several GPUs.