License YOLO
Back to Ultralytics Glossary

Tensor Parallelism

Tensor parallelism splits each layer's weight matrices across GPUs, which compute their slices in parallel and combine results with an all-reduce.

Tensor parallelism is a distributed training and inference technique that splits the weight matrices inside each layer of a neural network across several GPUs. Every GPU holds a slice of every layer, computes its share of each matrix multiplication at the same time as the others, and exchanges partial results so the layer's output comes out complete. It is also called intra-layer model parallelism. NVIDIA researchers Mohammad Shoeybi and colleagues popularized the technique for transformers in Megatron-LM in September 2019, using it with data parallelism to train an 8.3-billion-parameter GPT-2-style model on 512 GPUs at 76% scaling efficiency.

How Tensor Parallelism Splits a Layer#

A transformer's feed-forward block multiplies its input by two weight matrices with an activation function in between. Megatron-LM splits the first matrix by columns and the second by rows. Each GPU multiplies the full input by its column slice, applies the activation to its own slice of the hidden units with no communication, and multiplies the result by its matching row slice. The partial outputs then only need summing, which one all-reduce operation does across the GPUs. Self-attention splits the same way along its heads, since every head computes independently until the output projection. Simulating four GPUs on one machine shows the sharded layer gives the same answer as the full one:

import torch

torch.manual_seed(0)
tp, d_model, d_ff = 4, 512, 2048  # 4-way tensor parallelism
x = torch.randn(8, d_model)  # 8 token embeddings, replicated on every GPU
w1 = torch.randn(d_model, d_ff) / d_model**0.5
w2 = torch.randn(d_ff, d_model) / d_ff**0.5

# GPU i holds column slice i of the first weight and row slice i of the second
w1_shards, w2_shards = w1.chunk(tp, dim=1), w2.chunk(tp, dim=0)
partials = [torch.relu(x @ w1_i) @ w2_i for w1_i, w2_i in zip(w1_shards, w2_shards)]

# One all-reduce, a sum of the partial outputs across GPUs, completes the layer
output = sum(partials)
reference = torch.relu(x @ w1) @ w2
print(f"Weights per GPU: {w1_shards[0].numel() + w2_shards[0].numel():,} of {w1.numel() + w2.numel():,}")
print("Matches the unsharded MLP:", torch.allclose(output, reference, atol=1e-5))

It prints Weights per GPU: 524,288 of 2,097,152 and Matches the unsharded MLP: True: each simulated GPU stores a quarter of the model weights, and the summed partials equal the full layer. Real frameworks apply this pattern per module: PyTorch offers ColwiseParallel and RowwiseParallel styles in its tensor parallel API, built on distributed tensors.

Communication Cost and Where It Runs#

The split is exact but chatty. Each transformer layer needs two all-reduces in the forward pass, one after attention and one after the feed-forward block, and two more during backpropagation. Because these sit on the critical path of every layer, tensor parallelism usually stays within one server, where eight GPUs share fast NVLink connections. Meta trained Llama 3 405B with 8-way tensor parallelism inside each server, 16-way pipeline parallelism across servers, and data parallelism over the rest, reaching 41% to 43% of peak BF16 throughput on 8,192 and 16,384 H100 GPUs. Tensor parallelism also leaves activations such as layer norm outputs replicated on every GPU, which sequence parallelism fixes by splitting those along the token dimension. DeepSeek-V3 shows the cost is avoidable: its team trained the 671-billion-parameter mixture-of-experts model without tensor parallelism, relying on pipeline and expert parallelism plus memory savings.

Inference uses the same split. Serving engines such as vLLM take a tensor parallel size and shard a large language model (LLM) and its KV cache across GPUs, so a model too large for one card can run at all. Each GPU reads only its shard of the weights for every generated token, so their combined memory bandwidth cuts inference latency.

Tensor Parallelism vs. Other Parallelism#

  • Pipeline parallelism: Places whole layers on different GPUs and streams micro-batches through them. It communicates only at stage boundaries, so it suits slower links between servers, at the cost of idle time while the pipeline fills.
  • Fully sharded data parallelism (FSDP): Shards weights, gradients, and optimizer states across data-parallel workers, then gathers each layer's full weights just before computing it. Tensor parallelism never assembles a full weight matrix on any GPU.
  • Context parallelism: Splits a long input sequence across GPUs instead of the weights.
  • Expert parallelism: Places different experts of a mixture-of-experts layer on different GPUs.

Data parallelism, which copies the whole model to every GPU and splits each batch, covers every model that fits in one device's memory. Ultralytics YOLO26 is one: YOLO26x, its largest object detection model, trains with about 59 million parameters, roughly 236 MB of FP32 weights, so passing device=[0, 1, 2, 3] in training mode runs PyTorch DistributedDataParallel with a complete model copy on each GPU. Tensor parallelism pays off for far wider layers: each of the 126 layers of Llama 3 405B multiplies a 16,384-wide hidden state by feed-forward matrices with 53,248 hidden units, about 2.6 billion feed-forward parameters per layer.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning