Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network architecture that splits part of a model into many parallel subnetworks, called experts, and uses a small learned router to send each input to only a few of them. Because each token or image patch passes through a fraction of the experts, the total parameter count can grow far faster than the computation spent per input. Mixtral 8x7B has 47 billion parameters but uses 13 billion for each token, and DeepSeek-V3 activates 37 billion of its 671 billion. The router-gated, sparse MoE layer used in today's large language models (LLMs) builds on Google's 2017 sparsely-gated MoE layer, which reached 137 billion parameters between stacked LSTM layers.

How an MoE Layer Works#

In a transformer, an MoE layer usually replaces the feed-forward network (FFN) that follows each self-attention block. The layer holds N expert FFNs of identical shape plus a router, a single linear layer that scores every expert for each token. The token goes to the k highest-scoring experts, their outputs are weighted by the router's normalized scores and summed, and the remaining experts do no work for that token. Mixtral picks 2 of 8 experts in every layer, while DeepSeek-V3 picks 8 of 256 smaller routed experts and also sends every token through one shared expert. Because routing happens independently in every layer, a token can use different experts at different depths, and two tokens in the same sentence can take different paths through the same layer:

The experts in Mixtral did not specialize by topic. Its authors found no clear assignment of experts to subjects such as biology or philosophy; routing followed syntax instead, so tokens such as self in Python code and indentation tokens tended to reach the same experts.

Load Balancing and Expert Capacity#

A freshly initialized router tends to favor a few experts, which then train faster and attract even more tokens while the rest stay idle. The six-token example below routes through a top-2 layer with eight experts and counts how many tokens each expert receives:

import torch

torch.manual_seed(0)
dim, num_experts, top_k = 32, 8, 2
experts = torch.nn.ModuleList(
    torch.nn.Sequential(torch.nn.Linear(dim, 4 * dim), torch.nn.GELU(), torch.nn.Linear(4 * dim, dim))
    for _ in range(num_experts)
)
router = torch.nn.Linear(dim, num_experts, bias=False)
tokens = torch.randn(6, dim)

# The router keeps each token's top-2 expert scores and softmaxes them into mixing weights
scores, chosen = router(tokens).topk(top_k, dim=-1)
weights = scores.softmax(dim=-1)
out = torch.zeros_like(tokens)
for e, expert in enumerate(experts):
    token_idx, slot = (chosen == e).nonzero(as_tuple=True)  # tokens that picked expert e
    out[token_idx] += weights[token_idx, slot, None] * expert(tokens[token_idx])

print("Experts per token:", chosen.tolist())
print("Tokens per expert:", torch.bincount(chosen.flatten(), minlength=num_experts).tolist())

It prints the expert pairs [[3, 6], [7, 6], [0, 7], [3, 1], [6, 1], [2, 3]] and the per-expert counts [1, 2, 1, 3, 0, 0, 3, 2]: experts 3 and 6 each process three tokens while experts 4 and 5 receive none, and every token runs through only 2 of the 8 expert networks. The Switch Transformer, which routed each token to a single expert and scaled to 1.6 trillion parameters, added an auxiliary loss term that is smallest when tokens and router probability are spread evenly across experts, and many later MoE models use a similar term. DeepSeek-V3 drops the auxiliary loss and instead nudges a per-expert bias on the routing scores up or down during training. Training systems also cap the number of tokens an expert accepts per batch, its capacity; tokens over the cap skip that layer's experts and continue through the residual connection.

MoE vs. Related Concepts#

• Model ensemble: Several complete models each process every input and their predictions are averaged or voted, so compute grows with each model added. An MoE is one network trained end to end, with the router choosing a different path for each token.
• Model routing: A router picks one whole model per request, such as a small model for simple questions and a large one for hard ones. MoE routing happens per token inside every MoE layer of a single model.
• Mixture of depths (MoD): A router lets some tokens skip an entire block, so compute varies per token. It is often combined with MoE.
• Mixture of agents (MoA): Several separate LLMs answer a prompt and later layers of LLMs refine their combined answers, a prompting pipeline with no shared weights.

MoE in Language and Vision Models#

Many of the largest open-weight LLMs use MoE layers, including DeepSeek-V3, Llama 4, the MoE versions of Qwen3, and OpenAI's gpt-oss. In computer vision, Vision MoE (V-MoE) replaced some FFN layers of a Vision Transformer (ViT) with experts and routed individual image patches, matching state-of-the-art accuracy with as little as half the inference compute, and its 15-billion-parameter version reached 90.35% top-1 accuracy on ImageNet.

Memory and Serving#

MoE saves compute, not memory: every expert has to be loaded even though each token uses few of them, so Mixtral needs the memory of a 47-billion-parameter model while doing the per-token work of a 13-billion-parameter one. Large MoE models are therefore spread across GPUs with expert parallelism, where each GPU holds a subset of experts and an all-to-all exchange sends every token to the devices that hold its chosen experts and back. Frameworks such as DeepSpeed-MoE combine expert parallelism with data and tensor parallelism for training. Edge vision models take the dense route: YOLO26 runs every layer on every image, so its memory footprint and compute per image are the same for every input.