SwiGLU

SwiGLU (Swish-gated linear unit) is a gated variant of the feed-forward layer in transformers. It projects each token vector twice, passes one projection through the SiLU (Swish) activation function, multiplies the two element-wise, and projects the product back to the model width: FFN(x) = (SiLU(xW) ⊙ xV)W₂. Noam Shazeer introduced it in the 2020 paper GLU Variants Improve Transformer, and it has since become the feed-forward block of large language models (LLMs) such as PaLM, Llama, Mistral, and Qwen. Unlike ReLU or GELU, which act on each number independently, SwiGLU contains learned weights, which makes it a small layer of its own.

How SwiGLU Works#

A standard transformer feed-forward network (FFN) expands each token vector with one weight matrix, applies an activation, and compresses it back with a second matrix. SwiGLU replaces the first matrix and the activation with a gated linear unit (GLU), which has two parallel projections of the same input:

• Gate projection: xW passes through SiLU, x · σ(x), producing values that are near zero for strongly negative inputs and close to the input for positive ones.
• Value projection: xV stays linear and carries the content.
• Element-wise product: Multiplying the two lets the gate scale each hidden feature up, down, or to nearly zero depending on the input, and the output projection W₂ maps the result back to the model width.

The gating idea comes from the GLU of Dauphin and colleagues, who used a sigmoid gate in gated convolutional language models in 2016. Shazeer's paper tested the same structure with different gate functions and named each variant after its activation: GLU uses sigmoid, ReGLU uses ReLU, GEGLU uses GELU, SwiGLU uses Swish, and a bilinear layer uses no activation at all. Following the T5 codebase, these layers drop the bias terms.

Why SwiGLU Replaced ReLU and GELU in LLMs#

Gated feed-forward layers have three weight matrices instead of two, so Shazeer shrank the hidden width by a factor of 2/3 to keep parameters and computation equal, from 3,072 to 2,048 units in a T5-base model. At equal size, SwiGLU reached a held-out log-perplexity of 1.636 after 524,288 training steps, against 1.677 for ReLU and 1.679 for GELU, and it also posted higher average scores on the GLUE and SuperGLUE language-understanding benchmarks. The paper offers no theory for the gain; its conclusion attributes the success of these architectures, "as all else, to divine benevolence."

The result carried over to large models. Google's PaLM adopted SwiGLU in 2022, and Meta's LLaMA replaced ReLU with SwiGLU in 2023, using a hidden width of 2/3 × 4d rounded up to a multiple of 256, which gives 11,008 hidden units for the 4,096-dimensional 7B model. Llama 2, Llama 3, Mistral, and Qwen kept the design, and in mixture-of-experts (MoE) models such as Mixtral, each expert is a SwiGLU feed-forward block.

Implementing SwiGLU in PyTorch#

The PyTorch snippet below builds a standard ReLU feed-forward layer and a SwiGLU layer for a 512-dimensional model and compares their sizes:

import torch
import torch.nn.functional as F
from torch import nn

d_model = 512
x = torch.randn(4, d_model)  # four token vectors

# Standard transformer FFN: two matrices around a 4x wider ReLU hidden layer
relu_ffn = nn.Sequential(nn.Linear(d_model, 2048, bias=False), nn.ReLU(), nn.Linear(2048, d_model, bias=False))

# SwiGLU FFN: three matrices, hidden width cut to about 2/3 of 2048
w_gate = nn.Linear(d_model, 1365, bias=False)
w_value = nn.Linear(d_model, 1365, bias=False)
w_out = nn.Linear(1365, d_model, bias=False)
y = w_out(F.silu(w_gate(x)) * w_value(x))  # (SiLU(xW) * xV) W2

relu_params = sum(p.numel() for p in relu_ffn.parameters())
swiglu_params = sum(m.weight.numel() for m in (w_gate, w_value, w_out))
print(f"ReLU FFN:   output {tuple(relu_ffn(x).shape)}, {relu_params:,} parameters")
print(f"SwiGLU FFN: output {tuple(y.shape)}, {swiglu_params:,} parameters")

It prints ReLU FFN: output (4, 512), 2,097,152 parameters and SwiGLU FFN: output (4, 512), 2,096,640 parameters. With 1,365 hidden units, about 2/3 of 2,048, the three SwiGLU matrices hold almost exactly as many weights as the two ReLU matrices, so any quality difference comes from the gating. Many implementations fuse the gate and value matrices into one projection and split its output in half, which runs one larger matrix multiplication instead of two smaller ones.

SwiGLU vs. Related Terms#

• SiLU (Swish): SiLU is the element-wise function inside the gate. SwiGLU is the whole gated layer with its three projections.
• GLU and GEGLU: Sibling variants that use a sigmoid or a GELU gate. GEGLU matched SwiGLU in Shazeer's tests, with a log-perplexity of 1.633, and Google's Gemma models use it.
• Attention: Attention also produces data-dependent weights, but it mixes information across tokens. SwiGLU gates features within each token independently.
• Vision models: Gated feed-forward layers also appear in vision and multimodal models. YOLOE, the open-vocabulary detector from Tsinghua University researchers that runs in the Ultralytics package, refines the text embeddings of class prompts with a residual SwiGLU block that runs once per set_classes() call and is folded away at export, so it adds no cost per frame.