Context Parallelism
Context parallelism splits one long input sequence across GPUs that exchange keys and values during attention, cutting the memory each GPU needs.
Context parallelism (CP) is a distributed training technique that splits a single long input sequence across several GPUs. Each GPU holds a contiguous or interleaved share of the tokens and all activations computed from them, so the memory each device needs shrinks as devices are added. Layers that treat tokens independently, such as normalization and feed-forward blocks, run on local tokens with no communication. Self-attention is the exception, because every query may look at every key, so GPUs exchange keys and values to produce exactly the same output as attention over the full sequence. It is a systems technique for running a given context window, whose length is fixed by the model's architecture and training.
How GPUs Share Keys and Values#
Implementations differ mainly in how keys and values reach the GPUs that need them:
- Ring attention: GPUs form a ring and pass key-value blocks to their neighbors while computing on the block they hold, hiding transfers behind computation.
- All-gather: Every GPU collects all keys and values, then computes attention for its own queries. Meta chose this for Llama 3 because it supports document masks easily, and grouped-query attention keeps keys and values much smaller than queries.
- All-to-all: DeepSpeed Ulysses reshuffles the data so each GPU holds every token for a subset of attention heads, which caps the parallel degree at the number of heads.
PyTorch's experimental context parallel API swaps scaled dot-product attention for ring attention inside a context_parallel() block, and NVIDIA's Megatron-Core gathers keys and values over ring-shaped point-to-point transfers in the forward pass and reduce-scatters their gradients in the backward pass. CP differs from Megatron's older sequence parallelism, which splits only layer norm and dropout activations alongside tensor parallelism, because it splits the inputs and every activation along the sequence.
Balancing Causal Attention#
With a causal mask, token i attends only to the i tokens before it and itself, so later tokens cost more. Giving each GPU one contiguous quarter of the sequence leaves the first GPU with little work and the last with most of it, and the busiest GPU sets the pace. Llama 3 and Megatron-Core cut the sequence into twice as many chunks as GPUs and give each GPU one early chunk and its mirror from the end:
import torch
seq_len, ranks = 8192, 4
positions = torch.arange(seq_len)
causal_work = positions + 1 # under a causal mask, token i attends to i + 1 keys
# Contiguous split: rank r takes the r-th quarter of the sequence
contiguous = [causal_work[shard].sum().item() for shard in positions.chunk(ranks)]
# Load-balanced split: cut into 2 * ranks chunks, rank r takes chunk r and its mirror from the end
chunks = positions.chunk(2 * ranks)
balanced = [causal_work[torch.cat([chunks[r], chunks[-1 - r]])].sum().item() for r in range(ranks)]
print("Contiguous, millions of scores per rank:", [round(w / 1e6, 1) for w in contiguous])
print("Balanced, millions of scores per rank: ", [round(w / 1e6, 1) for w in balanced])The contiguous split gives the four GPUs [2.1, 6.3, 10.5, 14.7] million attention scores, a 7× spread, and the mirrored split gives each 8.4 million.
Where Context Parallelism Is Used#
Training large language models (LLMs) on long documents and codebases is the main use. Meta extended Llama 3 405B to 131,072-token sequences with 16-way context parallelism alongside tensor, pipeline, and data parallelism on 16,384 H100 GPUs, at 38% of peak BF16 throughput compared with 41% for the same cluster at 8,192 tokens. Video is the other: Meta's Movie Gen trained its 30-billion-parameter video generation model on clips encoded as roughly 73,000 tokens, using context parallelism with tensor, sequence, and fully sharded data parallelism. Long-video vision-language models and video understanding systems hit the same limit, because each frame adds hundreds of tokens.
Context parallelism combines with the other strategies:
- Pipeline parallelism: Splits the model's layers across GPUs.
- Data parallelism: Splits the batch across complete model copies.
- Flash attention: Avoids storing the attention matrix on one GPU. Context parallel implementations run a kernel like it on each GPU's local block.
Image models reach these lengths only at very high resolution, since sequence length grows with the number of patches or feature-map positions. Ultralytics YOLO26 places self-attention only on its coarsest, stride-32 feature map, in layers 10 and 22 of its model configuration, so attention covers 400 positions at the default imgsz=640 and 1,600 at imgsz=1280.










