ReLU (Rectified Linear Unit)

The rectified linear unit (ReLU) is the activation function f(x) = max(0, x): it passes positive inputs through unchanged and outputs zero for everything else. That single comparison gives a neural network (NN) the nonlinearity it needs to learn curved patterns, at almost no computational cost. ReLU became the default hidden-layer activation in deep learning after Vinod Nair and Geoffrey Hinton used it in 2010 and AlexNet, a convolutional neural network (CNN) built on ReLUs, won the 2012 ImageNet challenge. It remains the baseline that newer activations such as SiLU and GELU are measured against.

How ReLU Works#

ReLU's derivative is 1 for positive inputs and 0 for negative inputs. At exactly zero the derivative is undefined, and frameworks such as PyTorch use 0. Those two values explain ReLU's main properties:

• Cheap to compute: The forward pass is a comparison with zero and the backward pass is a mask, with no exponentials as in sigmoid or tanh.
• No saturation for positive inputs: A gradient of exactly 1 passes through every active unit unchanged, so error signals do not shrink as they travel backward through many layers, which counters the vanishing gradient problem that saturating functions cause.
• Sparse activations: Every negative input becomes an exact zero, so at any moment only part of a layer is active. With randomly initialized weights, about half of a layer's units output zero for a given input.
• Piecewise linear: A network built from ReLUs computes a piecewise linear function of its input, with many linear regions joined at the points where units switch on or off.

The PyTorch example below applies nn.ReLU and uses autograd to compute its gradient at the same points:

import torch
from torch import nn

relu = nn.ReLU()

# Inputs on both sides of zero, tracked so PyTorch can compute gradients
x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0], requires_grad=True)
y = relu(x)

# Backpropagate a gradient of 1 from each output to its input
y.sum().backward()

print(f"Output:   {y.tolist()}")
print(f"Gradient: {x.grad.tolist()}")

It prints the outputs [0.0, 0.0, 0.0, 0.5, 2.0] and the gradients [0.0, 0.0, 0.0, 1.0, 1.0]. Positive inputs pass through with their gradient intact, and the negative inputs and zero receive no gradient at all, which is the source of both ReLU's sparsity and its main failure mode.

The Dying ReLU Problem#

A ReLU unit whose input is negative for every example in the training data outputs zero and receives zero gradient, so its weights never change again and the unit is effectively dead. Units usually die when a large weight update, often caused by a high learning rate, pushes their bias far negative. A few dead units cost little, while a badly tuned run can kill a large share of a layer. He initialization, which scales initial weights for ReLU layers, together with moderate learning rates and batch normalization, keeps most units alive, and leaky variants remove the problem by never letting the gradient reach zero.

ReLU Variants#

• Leaky ReLU: Multiplies negative inputs by a small fixed slope, typically 0.01, so inactive units still receive a gradient.
• PReLU: Learns the negative slope during training. Kaiming He and colleagues introduced it in 2015 in the same paper that proposed He initialization.
• ReLU6: Caps the output at 6, min(max(0, x), 6), which keeps activations in a range that low-precision arithmetic represents well. MobileNetV1 and MobileNetV2 use it.
• ELU: Replaces the negative side with a smooth exponential curve that saturates at −1, pushing mean activations closer to zero.
• SiLU and GELU: Smooth curves, x · σ(x) and x · Φ(x), that behave like ReLU for large inputs but pass small negative values and have no kink at zero. They cost more per element, and the GELU paper reported gains over ReLU on every vision, language, and speech task it tested.

ReLU vs. Sigmoid and Tanh#

Before ReLU, hidden layers used sigmoid or tanh. Both saturate: for large positive or negative inputs their curves flatten and their derivatives approach zero, and sigmoid's derivative never exceeds 0.25 even at its steepest point. Multiplying such derivatives across many layers starves early layers of gradient. In the AlexNet paper, a four-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh units. Sigmoid and tanh survive where their bounded ranges are the point: sigmoid in output layers for probabilities and in the gates of recurrent cells, and tanh in recurrent hidden states.

ReLU in Modern Models#

ResNet and most CNNs of its era use ReLU throughout, and the original 2017 transformer used it in its feed-forward layers. Many newer models switched to smoother functions, GELU in BERT and vision transformers and SwiGLU in many current large language models, while ReLU stays common where speed and hardware support matter most. YOLO26 uses SiLU in its Conv blocks by default, and an activation: nn.ReLU() line in the model YAML swaps ReLU into every Conv block before training, which suits accelerators such as the AMD Zynq UltraScale+ DPU that run ReLU natively but not SiLU. In PyTorch, nn.ReLU(inplace=True) overwrites its input tensor with the result, saving the memory of a separate output tensor.