GELU (Gaussian Error Linear Unit)

The Gaussian error linear unit (GELU) is an activation function defined as GELU(x) = x · Φ(x), where Φ is the cumulative distribution function of the standard normal distribution. It weights each input by how likely a standard normal variable is to fall below it, so large positive inputs pass almost unchanged, large negative inputs go to nearly zero, and inputs near zero are scaled by about half. Dan Hendrycks and Kevin Gimpel introduced it in the 2016 paper Gaussian Error Linear Units (GELUs). GELU became the standard activation of transformer models such as BERT, GPT-2, GPT-3, and the vision transformer (ViT), where it sits between the two linear layers of every feed-forward block.

How GELU Works#

Hendrycks and Gimpel derived GELU from stochastic regularization. Picture multiplying each input x by a random mask that keeps it with probability Φ(x) and zeroes it otherwise, like dropout with a keep probability that grows with the input. GELU is the expected value of that operation, Φ(x) · x, so it gates inputs smoothly by their value, where ReLU gates them abruptly by their sign.

The resulting curve has three properties that matter for training:

• Smooth everywhere: GELU and its derivative are continuous, with no kink at zero, so small weight changes produce small output changes during backpropagation.
• Non-monotonic: For negative inputs, the output first dips and then rises back toward zero, reaching a minimum of about −0.17 at x ≈ −0.75.
• Gradient for negative inputs: Units with moderately negative inputs still receive a gradient, so they keep learning where ReLU units would stop.

Because Φ involves the error function, two approximations are common. The tanh form, 0.5x(1 + tanh(√(2/π)(x + 0.044715x³))), stays within about 0.0005 of exact GELU and was used in GPT-2. The sigmoid form, x · σ(1.702x), is within about 0.02 and is the QuickGELU used in OpenAI's CLIP. PyTorch's nn.GELU computes the exact form by default and the tanh form with approximate="tanh":

import torch
from torch import nn

x = torch.tensor([-3.0, -1.0, -0.75, 0.0, 1.0, 3.0])

# Exact GELU: x * Phi(x), where Phi is the standard normal CDF
exact = nn.GELU()(x)

# Tanh approximation, as used in GPT-2
approx = nn.GELU(approximate="tanh")(x)

print(f"GELU exact: {[round(v, 4) for v in exact.tolist()]}")
print(f"GELU tanh:  {[round(v, 4) for v in approx.tolist()]}")
print(f"ReLU:       {nn.ReLU()(x).tolist()}")

Exact GELU prints [-0.004, -0.1587, -0.17, 0.0, 0.8413, 2.996], the tanh form [-0.0036, -0.1588, -0.17, 0.0, 0.8412, 2.9964], and ReLU [0.0, 0.0, 0.0, 0.0, 1.0, 3.0]. GELU outputs −0.17 at its minimum near −0.75, maps 1 to 0.8413 because Φ(1) = 0.8413, and nearly matches ReLU at ±3. The two GELU forms differ only in the fourth decimal place.

Where GELU Is Used#

• GPT models: OpenAI's first GPT used GELU in 2018, and GPT-2 and GPT-3 kept it.
• BERT: Google's BERT chose GELU over ReLU following GPT, and most BERT-style encoders still use it.
• Vision transformers: ViT applies GELU in the MLP of every encoder block, and many vision backbones followed, including ConvNeXt, which replaced ReLU with GELU while modernizing ResNet.
• Detection transformers: In the Ultralytics package, the AIFI transformer encoder of RT-DETR, Baidu's real-time detection transformer, applies GELU between its two feed-forward layers.

Many recent large language models (LLMs) moved from plain GELU to gated feed-forward layers. Google's Gemma uses GEGLU, which multiplies a GELU-activated projection by a second linear projection, and Llama, PaLM, and Qwen use the SiLU-based SwiGLU.

GELU vs. Related Activations#

• ReLU: max(0, x) is cheaper, outputs exact zeros, and has a kink at zero. The GELU paper reported gains over ReLU and ELU on every computer vision, natural language processing, and speech task it tested.
• SiLU (Swish): x · σ(x) has the same overall shape with a deeper minimum of about −0.28. GELU is the usual choice in transformers, and SiLU is more common in convolutional detectors.
• Leaky ReLU: Keeps a fixed straight-line slope for negative inputs, where GELU's negative side is a curve that returns to zero.
• GEGLU: The gated variant adds a second learned projection. In Noam Shazeer's 2020 comparison on T5, GEGLU feed-forward layers reached a held-out log-perplexity of 1.633, against 1.679 for GELU and 1.677 for ReLU at the same parameter count.