Tensor Parallelism
Tensor parallelism splits each layer's weight matrices across GPUs, which compute their slices in parallel and combine results with an all-reduce.
Tensor parallelism is a distributed training and inference technique that splits the weight matrices inside each layer of a neural network across several GPUs. Every GPU holds a slice of every layer, computes its share of each matrix multiplication at the same time as the others, and exchanges partial results so the layer's output comes out complete. It is also called intra-layer model parallelism. NVIDIA researchers Mohammad Shoeybi and colleagues popularized the technique for transformers in Megatron-LM in September 2019, using it with data parallelism to train an 8.3-billion-parameter GPT-2-style model on 512 GPUs at 76% scaling efficiency.
How Tensor Parallelism Splits a Layer#
A transformer's feed-forward block multiplies its input by two weight matrices with an activation function in between. Megatron-LM splits the first matrix by columns and the second by rows. Each GPU multiplies the full input by its column slice, applies the activation to its own slice of the hidden units with no communication, and multiplies the result by its matching row slice. The partial outputs then only need summing, which one all-reduce operation does across the GPUs. Self-attention splits the same way along its heads, since every head computes independently until the output projection. Simulating four GPUs on one machine shows the sharded layer gives the same answer as the full one:
import torch
torch.manual_seed(0)
tp, d_model, d_ff = 4, 512, 2048 # 4-way tensor parallelism
x = torch.randn(8, d_model) # 8 token embeddings, replicated on every GPU
w1 = torch.randn(d_model, d_ff) / d_model**0.5
w2 = torch.randn(d_ff, d_model) / d_ff**0.5
# GPU i holds column slice i of the first weight and row slice i of the second
w1_shards, w2_shards = w1.chunk(tp, dim=1), w2.chunk(tp, dim=0)
partials = [torch.relu(x @ w1_i) @ w2_i for w1_i, w2_i in zip(w1_shards, w2_shards)]
# One all-reduce, a sum of the partial outputs across GPUs, completes the layer
output = sum(partials)
reference = torch.relu(x @ w1) @ w2
print(f"Weights per GPU: {w1_shards[0].numel() + w2_shards[0].numel():,} of {w1.numel() + w2.numel():,}")
print("Matches the unsharded MLP:", torch.allclose(output, reference, atol=1e-5))It prints Weights per GPU: 524,288 of 2,097,152 and Matches the unsharded MLP: True: each simulated GPU stores a quarter of the model weights, and the summed partials equal the full layer. Real frameworks apply this pattern per module: PyTorch offers ColwiseParallel and RowwiseParallel styles in its tensor parallel API, built on distributed tensors.
Communication Cost and Where It Runs#
The split is exact but chatty. Each transformer layer needs two all-reduces in the forward pass, one after attention and one after the feed-forward block, and two more during backpropagation. Because these sit on the critical path of every layer, tensor parallelism usually stays within one server, where eight GPUs share fast NVLink connections. Meta trained Llama 3 405B with 8-way tensor parallelism inside each server, 16-way pipeline parallelism across servers, and data parallelism over the rest, reaching 41% to 43% of peak BF16 throughput on 8,192 and 16,384 H100 GPUs. Tensor parallelism also leaves activations such as layer norm outputs replicated on every GPU, which sequence parallelism fixes by splitting those along the token dimension. DeepSeek-V3 shows the cost is avoidable: its team trained the 671-billion-parameter mixture-of-experts model without tensor parallelism, relying on pipeline and expert parallelism plus memory savings.
Inference uses the same split. Serving engines such as vLLM take a tensor parallel size and shard a large language model (LLM) and its KV cache across GPUs, so a model too large for one card can run at all. Each GPU reads only its shard of the weights for every generated token, so their combined memory bandwidth cuts inference latency.
Tensor Parallelism vs. Other Parallelism#
- Pipeline parallelism: Places whole layers on different GPUs and streams micro-batches through them. It communicates only at stage boundaries, so it suits slower links between servers, at the cost of idle time while the pipeline fills.
- Fully sharded data parallelism (FSDP): Shards weights, gradients, and optimizer states across data-parallel workers, then gathers each layer's full weights just before computing it. Tensor parallelism never assembles a full weight matrix on any GPU.
- Context parallelism: Splits a long input sequence across GPUs instead of the weights.
- Expert parallelism: Places different experts of a mixture-of-experts layer on different GPUs.
Data parallelism, which copies the whole model to every GPU and splits each batch, covers every model that fits in one device's memory. Ultralytics YOLO26 is one: YOLO26x, its largest object detection model, trains with about 59 million parameters, roughly 236 MB of FP32 weights, so passing device=[0, 1, 2, 3] in training mode runs PyTorch DistributedDataParallel with a complete model copy on each GPU. Tensor parallelism pays off for far wider layers: each of the 126 layers of Llama 3 405B multiplies a 16,384-wide hidden state by feed-forward matrices with 53,248 hidden units, about 2.6 billion feed-forward parameters per layer.










