Activation Function

An activation function is the function a neuron in a neural network (NN) applies to the weighted sum of its inputs to produce its output. Its essential job is to add nonlinearity: without it, any stack of layers collapses into a single linear transformation, and depth adds nothing. With it, a network can represent curved decision boundaries, from the strokes of handwritten digits to the outline of a pedestrian in a camera frame. Activation functions play two roles. In hidden layers, functions such as ReLU, SiLU, and GELU let the network build features. In the output layer, functions such as sigmoid and softmax turn raw scores into probabilities.

Why Networks Need Nonlinearity#

A layer without an activation computes Wx + b. Two such layers give W₂(W₁x + b₁) + b₂, which equals (W₂W₁)x + (W₂b₁ + b₂), another single linear layer. An activation between the layers breaks this collapse, and the universal approximation theorem shows that one hidden layer with enough nonlinear units can approximate any continuous function on a bounded domain.

The activation's derivative matters as much as its shape, because backpropagation multiplies these derivatives layer by layer. Sigmoid's derivative never exceeds 0.25, and tanh's falls toward zero for large inputs, so gradients shrink as they pass through many saturating layers, the vanishing gradient problem. ReLU's derivative is exactly 1 for every positive input, which is a main reason it made very deep networks practical to train.

Common Activation Functions#

• ReLU: max(0, x). Cheap, with a gradient of 1 for positive inputs and exact zeros for negative ones. It has been the default hidden activation of CNNs since AlexNet in 2012, but units that only ever see negative inputs stop learning.
• Leaky ReLU: Multiplies negative inputs by a small slope such as 0.01 or 0.1, so every unit keeps a gradient.
• SiLU (Swish): x · σ(x), a smooth curve that dips slightly below zero for negative inputs. It is common in efficient CNNs and real-time detectors.
• GELU: x · Φ(x), where Φ is the standard normal cumulative distribution function. It is the standard activation in BERT, GPT-2, and vision transformers.
• Tanh: Squashes inputs into −1 to 1 around a zero center. It shapes the hidden state of recurrent networks such as the GRU.
• Sigmoid: Squashes inputs into 0 to 1, which suits binary and multi-label outputs and the gates of recurrent cells.
• Softmax: Turns a whole vector of scores into a probability distribution that sums to 1, for single-label classification and attention weights.
• SwiGLU: A gated layer that multiplies a SiLU-activated projection by a second learned projection. It replaced the plain activation in the feed-forward layers of many recent large language models (LLMs).

PyTorch provides each element-wise function as a module in torch.nn, so they are easy to compare on the same inputs:

import torch
from torch import nn

x = torch.tensor([-3.0, -1.0, 0.0, 1.0, 3.0])

# Common activations, each applied element-wise to the same inputs
activations = {
    "ReLU": nn.ReLU(),
    "Leaky ReLU": nn.LeakyReLU(0.1),
    "SiLU": nn.SiLU(),
    "GELU": nn.GELU(),
    "Tanh": nn.Tanh(),
    "Sigmoid": nn.Sigmoid(),
}
for name, act in activations.items():
    print(f"{name:>10}: {[round(v, 3) for v in act(x).tolist()]}")

For the inputs −3, −1, 0, 1, and 3, ReLU prints [0.0, 0.0, 0.0, 1.0, 3.0] and Leaky ReLU [-0.3, -0.1, 0.0, 1.0, 3.0]. SiLU gives [-0.142, -0.269, 0.0, 0.731, 2.858] and GELU [-0.004, -0.159, 0.0, 0.841, 2.996]: both let small negative values through and approach ReLU for large inputs. Tanh gives [-0.995, -0.762, 0.0, 0.762, 0.995] and sigmoid [0.047, 0.269, 0.5, 0.731, 0.953], and both flatten out at ±3, where their gradients are already small.

Choosing an Activation Function#

Hidden layers usually follow the convention of the architecture family, and the output layer follows the task. Softmax suits tasks where each input belongs to exactly one class, sigmoid suits labels that can co-occur, and regression outputs use no activation at all so they can take any value. Hardware can also constrain the choice: some edge accelerators natively run only simple functions such as ReLU, Leaky ReLU, and Hard-Swish. YOLO26 applies SiLU after the convolution and batch normalization in every Conv block, scores each detection class with an independent sigmoid so a box's class scores need not sum to 1, and applies softmax across the 1,000 ImageNet classes in its classification models.

Activation Functions vs. Related Terms#

• Loss function: The activation shapes each neuron's output during the forward pass, and the loss scores the network's final prediction against the label. Libraries often fuse the last activation into the loss, as CrossEntropyLoss does with softmax, for numerical stability.
• Batch normalization: Batch or layer normalization rescales activations to a stable range using statistics and learned parameters. It sits next to the activation function, usually just before it, and adds no nonlinearity of its own.
• Activations: As a plural noun, activations are the output values a layer produces. "Nonlinearity" and the older "transfer function" are synonyms for activation function. During training, frameworks keep every layer's activations in memory for the backward pass, which is why gradient checkpointing saves memory by discarding some activations and recomputing them during backpropagation.