Expert Parallelism

Expert parallelism (EP) is a distributed training and inference strategy for mixture-of-experts (MoE) models that places different experts on different GPUs and sends each token to the devices holding the experts chosen for it. An MoE layer replaces one large feed-forward block of a transformer with many smaller expert networks plus a router that activates only a few experts per token, so total parameters can grow much faster than the compute spent on each token. Expert parallelism is what makes those parameters fit in memory: instead of every GPU storing every expert, each stores a slice. Noam Shazeer and colleagues at Google already spread experts across devices, one shared copy each, in their 2017 sparsely gated MoE layer, and GShard scaled the approach in June 2020, spreading the experts of a translation model with more than 600 billion parameters across 2,048 TPU v3 accelerators. Expert parallelism is now standard for training and serving large MoE language and vision-language models.

How Tokens Move Between GPUs#

Each MoE layer runs in four steps:

1. Route: The router scores every token against every expert and keeps the top-k experts, commonly one, two, or eight.
2. Dispatch: An all-to-all collective sends each token's hidden state to the GPUs that own its chosen experts.
3. Compute: Each GPU runs its local experts on the tokens it received.
4. Combine: A second all-to-all returns the outputs to each token's original GPU, where they are weighted by the router scores and summed.

The snippet below simulates the routing and dispatch steps for one layer with 8 experts spread across 4 GPUs, two experts per GPU:

import torch

torch.manual_seed(0)
num_experts, num_gpus, top_k = 8, 4, 2
tokens = torch.randn(4096, 512)  # hidden states for 4,096 tokens
router = torch.nn.Linear(512, num_experts)

# Route: each token picks its top-2 experts by router score
with torch.no_grad():
    _, experts = router(tokens).topk(top_k, dim=-1)

# Expert parallelism: experts 0-1 live on GPU 0, experts 2-3 on GPU 1, and so on
gpu = experts // (num_experts // num_gpus)

# Dispatch: count the token-expert assignments each expert and GPU receives
print(torch.bincount(experts.flatten(), minlength=num_experts).tolist())
print(torch.bincount(gpu.flatten(), minlength=num_gpus).tolist())

The first line of output shows each expert's load, from 948 to 1,087 assignments against an even share of 1,024, and the second shows the four GPUs receiving between 1,968 and 2,139 of the 8,192 assignments. Even a randomly initialized router spreads work unevenly, and trained routers develop popular experts that skew the load far more.

Load Balancing and Capacity#

A layer finishes only when its busiest GPU does, so routing imbalance turns directly into idle hardware. MoE training counters it in three ways:

• Auxiliary balancing loss: An extra term in the loss function rewards the router for spreading tokens evenly across experts.
• Capacity factor: Each expert accepts at most a fixed number of tokens per batch, for example 1.25 times the even share, and overflow tokens skip that expert.
• Bias-based balancing: DeepSeek-V3 adds a per-expert bias to the router scores that only affects which experts are picked, lowering it for overloaded experts and raising it for idle ones after each step, which balances load without an auxiliary loss competing with the main objective.

With a capacity limit, each token's path depends on how busy its experts are:

A skipped token continues through the layer's residual connection unchanged, so heavy dropping hurts quality. DeepSeek-V3 drops no tokens during training, relying on its bias-based balancing instead.

Communication is the other cost. The two all-to-all exchanges happen in every MoE layer, so expert parallelism runs best within a node, where GPUs share fast NVLink connections, and becomes expensive over slower links between nodes. DeepSeek-V3 spreads its 256 routed experts per layer with 64-way expert parallelism over 8 nodes, sends each token to experts on at most 4 nodes, and overlaps the exchanges with computation.

Combining Parallelism Strategies#

Expert parallelism is one of several ways to split training across devices:

• Data parallelism: Replicates the full model on every GPU and splits each batch, synchronizing gradients after every step.
• Tensor parallelism: Splits individual weight matrices across GPUs, so every GPU takes part in every layer.
• Pipeline parallelism: Assigns consecutive layers to different GPUs and streams micro-batches through them.
• Context parallelism: Splits one long input sequence across GPUs.

Dense models that fit on one GPU need nothing beyond data parallelism, which is how Ultralytics YOLO26 trains on several GPUs when a list of devices such as device=[0, 1] is passed in training mode. Large MoE models combine several: DeepSeek-V3 trained its 671 billion parameters, 37 billion active per token, with 16-way pipeline parallelism, 64-way expert parallelism, and sharded data parallelism, and without tensor parallelism. Inference engines such as vLLM apply the same split when serving a model, so an MoE model whose experts would overflow one GPU's memory runs across several, each holding only its share of the experts.