License YOLO
Back to Ultralytics Glossary

Context Parallelism

Context parallelism splits one long input sequence across GPUs that exchange keys and values during attention, cutting the memory each GPU needs.

Context parallelism (CP) is a distributed training technique that splits a single long input sequence across several GPUs. Each GPU holds a contiguous or interleaved share of the tokens and all activations computed from them, so the memory each device needs shrinks as devices are added. Layers that treat tokens independently, such as normalization and feed-forward blocks, run on local tokens with no communication. Self-attention is the exception, because every query may look at every key, so GPUs exchange keys and values to produce exactly the same output as attention over the full sequence. It is a systems technique for running a given context window, whose length is fixed by the model's architecture and training.

How GPUs Share Keys and Values#

Implementations differ mainly in how keys and values reach the GPUs that need them:

  • Ring attention: GPUs form a ring and pass key-value blocks to their neighbors while computing on the block they hold, hiding transfers behind computation.
  • All-gather: Every GPU collects all keys and values, then computes attention for its own queries. Meta chose this for Llama 3 because it supports document masks easily, and grouped-query attention keeps keys and values much smaller than queries.
  • All-to-all: DeepSpeed Ulysses reshuffles the data so each GPU holds every token for a subset of attention heads, which caps the parallel degree at the number of heads.

PyTorch's experimental context parallel API swaps scaled dot-product attention for ring attention inside a context_parallel() block, and NVIDIA's Megatron-Core gathers keys and values over ring-shaped point-to-point transfers in the forward pass and reduce-scatters their gradients in the backward pass. CP differs from Megatron's older sequence parallelism, which splits only layer norm and dropout activations alongside tensor parallelism, because it splits the inputs and every activation along the sequence.

Balancing Causal Attention#

With a causal mask, token i attends only to the i tokens before it and itself, so later tokens cost more. Giving each GPU one contiguous quarter of the sequence leaves the first GPU with little work and the last with most of it, and the busiest GPU sets the pace. Llama 3 and Megatron-Core cut the sequence into twice as many chunks as GPUs and give each GPU one early chunk and its mirror from the end:

import torch

seq_len, ranks = 8192, 4
positions = torch.arange(seq_len)
causal_work = positions + 1  # under a causal mask, token i attends to i + 1 keys

# Contiguous split: rank r takes the r-th quarter of the sequence
contiguous = [causal_work[shard].sum().item() for shard in positions.chunk(ranks)]

# Load-balanced split: cut into 2 * ranks chunks, rank r takes chunk r and its mirror from the end
chunks = positions.chunk(2 * ranks)
balanced = [causal_work[torch.cat([chunks[r], chunks[-1 - r]])].sum().item() for r in range(ranks)]

print("Contiguous, millions of scores per rank:", [round(w / 1e6, 1) for w in contiguous])
print("Balanced, millions of scores per rank:  ", [round(w / 1e6, 1) for w in balanced])

The contiguous split gives the four GPUs [2.1, 6.3, 10.5, 14.7] million attention scores, a 7× spread, and the mirrored split gives each 8.4 million.

Where Context Parallelism Is Used#

Training large language models (LLMs) on long documents and codebases is the main use. Meta extended Llama 3 405B to 131,072-token sequences with 16-way context parallelism alongside tensor, pipeline, and data parallelism on 16,384 H100 GPUs, at 38% of peak BF16 throughput compared with 41% for the same cluster at 8,192 tokens. Video is the other: Meta's Movie Gen trained its 30-billion-parameter video generation model on clips encoded as roughly 73,000 tokens, using context parallelism with tensor, sequence, and fully sharded data parallelism. Long-video vision-language models and video understanding systems hit the same limit, because each frame adds hundreds of tokens.

Context parallelism combines with the other strategies:

  • Pipeline parallelism: Splits the model's layers across GPUs.
  • Data parallelism: Splits the batch across complete model copies.
  • Flash attention: Avoids storing the attention matrix on one GPU. Context parallel implementations run a kernel like it on each GPU's local block.

Image models reach these lengths only at very high resolution, since sequence length grows with the number of patches or feature-map positions. Ultralytics YOLO26 places self-attention only on its coarsest, stride-32 feature map, in layers 10 and 22 of its model configuration, so attention covers 400 positions at the default imgsz=640 and 1,600 at imgsz=1280.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning