SiLU (Sigmoid Linear Unit)

The Sigmoid Linear Unit (SiLU) is an activation function defined as SiLU(x) = x · σ(x), where σ is the sigmoid function. It passes large positive inputs almost unchanged, maps large negative inputs to nearly zero, and bends smoothly between the two, dipping to a minimum of about −0.278 at x ≈ −1.278. SiLU is the same function as Swish with β = 1, and the two names are used interchangeably. It is a common default in convolutional vision models, and inside the SwiGLU block it also drives the feed-forward layers of many large language models (LLMs).

Definition and Properties#

SiLU multiplies each input by a gate, σ(x), that lies between 0 and 1 and is computed from the input itself, which is why it is called self-gated. Its derivative is σ(x) · (1 + x · (1 − σ(x))), and together they give SiLU four properties that set its training behavior apart from ReLU:

• Smooth everywhere: The curve and its derivative are continuous, with no kink at zero, so small weight changes produce small output changes, which suits gradient descent.
• Gradient for negative inputs: A unit whose input is negative still receives a small gradient, so it keeps learning. ReLU units whose inputs stay negative receive no gradient at all and can stop learning permanently, a failure known as dying ReLU.
• Non-monotonic: The output falls slightly as x rises from large negative values to about −1.28 and rises after that, so the function is not monotonic.
• Bounded below, unbounded above: Outputs never drop below about −0.278, and positive inputs pass through without saturating, as with ReLU, which avoids the vanishing gradient behavior of sigmoid and tanh in deep networks.

The PyTorch code below evaluates SiLU and the gradients of SiLU and ReLU at five points.

import torch
import torch.nn.functional as F

x = torch.tensor([-4.0, -1.28, 0.0, 1.0, 4.0], requires_grad=True)

# SiLU(x) = x * sigmoid(x), compared with ReLU on the same inputs
silu = F.silu(x)
relu = F.relu(x)

# Gradient of each activation with respect to its input
silu_grad = torch.autograd.grad(silu.sum(), x)[0]
relu_grad = torch.autograd.grad(relu.sum(), x)[0]

print("SiLU output:  ", [round(v, 4) for v in silu.tolist()])
print("SiLU gradient:", [round(v, 4) for v in silu_grad.tolist()])
print("ReLU gradient:", relu_grad.tolist())

It prints SiLU outputs of [-0.0719, -0.2785, 0.0, 0.7311, 3.9281], SiLU gradients of [-0.0527, -0.0003, 0.5, 0.9277, 1.0527], and ReLU gradients of [0.0, 0.0, 0.0, 1.0, 1.0]. At −4, ReLU outputs 0 and passes no gradient, and SiLU still passes −0.0527. At −1.28, SiLU sits at its minimum of −0.2785 with a gradient of almost exactly 0, and at 4 its gradient slightly exceeds 1.

Origin of the Name#

Hendrycks and Gimpel named x · σ(x) the Sigmoid Linear Unit in their 2016 paper introducing GELU, where it appeared as a sigmoid-based relative of GELU that performed worse than GELU but usually better than ReLU and ELU. Elfwing, Uchibe, and Doya then used SiLU for neural network function approximation in reinforcement learning in 2017. Later that year, Google researchers rediscovered x · σ(βx) through an automated search for activation functions, named it Swish, and reported that replacing ReLU with it raised ImageNet top-1 accuracy by 0.9% for Mobile NASNet-A and by 0.6% for Inception-ResNet-v2. PyTorch added nn.SiLU in version 1.7 under the earlier name.

SiLU vs. Other Activation Functions#

• ReLU: max(0, x) is cheaper to compute, produces exact zeros, and has a constant gradient of 1 for positive inputs. SiLU adds one sigmoid per element in exchange for smoothness and gradients on the negative side.
• GELU: x · Φ(x), where Φ is the standard normal cumulative distribution function. The two curves have the same shape, and the scaled SiLU x · σ(1.702x) approximates GELU to within about 0.02 everywhere. GELU is the usual choice in transformers such as BERT and ViT, and SiLU is more common in CNN-based detectors.
• Sigmoid: σ(x) alone squashes inputs into the range 0 to 1 and saturates at both ends, so it serves as an output layer for probabilities. SiLU uses it only as the gate.
• Hard-Swish: x · ReLU6(x + 3) / 6, an approximation of SiLU introduced with MobileNetV3 that replaces the sigmoid gate with a piecewise-linear one, avoiding the exponential on mobile hardware.
• SwiGLU: A gated linear unit that multiplies SiLU(xW) element-wise by a second projection xV. It forms the feed-forward block of LLaMA and many other LLMs.

Where SiLU Is Used#

SiLU, under the Swish name, is the activation throughout EfficientNet, and it is common in modern real-time object detectors. Ultralytics YOLO models have used SiLU since YOLOv5 v4.0 replaced LeakyReLU and Hardswish with it in January 2021, and in YOLO26 every Conv block is a convolution, batch normalization, and SiLU.