Pipeline Parallelism

Pipeline parallelism is a distributed training technique that splits a neural network by depth into consecutive groups of layers called stages, places each stage on its own GPU, and passes activations from one stage to the next. To keep every GPU busy, each batch is cut into smaller micro-batches that flow through the stages like parts on an assembly line. It is also called inter-layer model parallelism. Google researchers led by Yanping Huang popularized the micro-batch approach with GPipe in November 2018, training a 557-million-parameter AmoebaNet to 84.4% top-1 accuracy on ImageNet and a 6-billion-parameter, 128-layer multilingual translation transformer.

Micro-Batches and the Pipeline Bubble#

With one batch at a time, only one stage works while the others wait: stage 2 cannot start until stage 1 finishes, and during backpropagation the order reverses. Micro-batches let stage 1 start the second micro-batch as soon as it hands the first to stage 2. The idle slots that remain while the pipeline fills and drains are called the bubble. Printing a GPipe-style forward schedule for 4 stages and 6 micro-batches shows its shape:

stages, micro_batches = 4, 6
steps = micro_batches + stages - 1  # time steps until the last micro-batch leaves the last stage

for stage in range(stages):
    row = ""
    for t in range(steps):
        m = t - stage  # stage s starts micro-batch m once stage s-1 hands it over
        row += f" F{m}" if 0 <= m < micro_batches else "  ."
    print(f"GPU {stage}:{row}")

# Every stage idles for stages - 1 of the steps, in the forward and again in the backward pass
bubble = (stages - 1) / steps
print(f"Idle fraction: {bubble:.0%}")

The output is a staircase: GPU 0 runs forward passes F0 to F5 in the first six of nine steps and then waits, GPU 3 waits three steps before starting F0, and the last line reads Idle fraction: 33%. The bubble fraction is (stages − 1) / (micro-batches + stages − 1), so 24 micro-batches through the same 4 stages cut it to 11%. Gradients from all micro-batches add up before a single optimizer step, the same arithmetic as gradient accumulation, so the update matches training on the full batch. GPipe also discards activations inside each stage and recomputes them during the backward pass, a form of gradient checkpointing.

Pipeline Schedules#

Later schedules reorder the same work to save memory or shrink the bubble:

• 1F1B: Introduced by the PipeDream system in 2018, it alternates one forward and one backward micro-batch per stage once the pipeline is full. Each stage then holds activations for at most as many micro-batches as there are stages, instead of all of them.
• Interleaved 1F1B: Gives each GPU several smaller, non-adjacent stages. NVIDIA's Megatron-LM team reported over 10% higher throughput with it and trained a 1-trillion-parameter model at 502 petaFLOP/s on 3,072 GPUs, 52% of peak.
• Zero bubble: Splits the backward pass into the gradient for the stage's input, which the previous stage is waiting for, and the gradient for the weights, which can fill idle slots later. It beats 1F1B throughput by up to 23% under a similar memory budget.

PyTorch provides these as ScheduleGPipe, Schedule1F1B, ScheduleInterleaved1F1B, ScheduleInterleavedZeroBubble, and ScheduleDualPipeV in its pipelining API, which also splits a model into stages. DualPipe comes from DeepSeek-V3, whose team ran 16 pipeline stages and used it to overlap computation with communication and shrink the bubble.

Where Pipeline Parallelism Fits#

Stages exchange only the activations at their boundaries, once per micro-batch, so pipeline parallelism tolerates slower links between servers better than tensor parallelism, which communicates inside every layer. Large training runs therefore combine them. Meta trained Llama 3 405B with 8-way tensor parallelism inside each 8-GPU server, 16 pipeline stages across servers, and data parallelism over 64 to 128 replicas, and serving engines such as vLLM use the same split when a large language model (LLM) needs more than one node. The slowest stage sets the pace for the whole pipeline, and the first and last stages of a language model also carry the embedding table and the output layer, so Llama 3 gave each of them one fewer transformer layer.

Related approaches split the work differently:

• Tensor parallelism: Splits the weight matrices inside each layer, so every GPU takes part in every layer.
• Context parallelism: Splits a long input sequence across GPUs.
• Split learning: Also cuts a network by depth, but between organizations, so that a data owner never sends raw data to the server running the later layers.

Data parallelism keeps every layer on every GPU and splits the batch instead, which is all a model needs when it fits on one device. That is how Ultralytics YOLO26 trains on several GPUs: with device=[0, 1, 2, 3] and batch=64 in training mode, each GPU runs the complete object detection model on 16 images, and PyTorch DistributedDataParallel averages the gradients across GPUs after every backward pass.