Leaky ReLU

Leaky ReLU (leaky rectified linear unit) is an activation function that returns its input unchanged when the input is positive and multiplies it by a small fixed slope, such as 0.01, when it is negative. It modifies the standard ReLU, which outputs zero for every negative input, so that negative values still produce a small output and, more importantly, a nonzero gradient during backpropagation. Andrew Maas, Awni Hannun, and Andrew Ng described the function in a 2013 study of rectifier activations in speech recognition acoustic models, using a slope of 0.01. It is used in deep learning models where gradient flow matters, from generative adversarial network discriminators to older YOLO object detectors and neural processing units that run quantized models with piecewise-linear activations.

How Leaky ReLU Works#

Leaky ReLU computes f(x) = x for x > 0 and f(x) = αx for x ≤ 0, where the negative slope α is a fixed hyperparameter between 0 and 1. Its derivative is 1 for positive inputs and α for negative inputs, so every input passes some gradient back to the weights that produced it. The usual slope differs by source and framework: PyTorch's nn.LeakyReLU defaults to 0.01, TensorFlow's tf.nn.leaky_relu to 0.2, and the Keras LeakyReLU layer to 0.3. This code passes negative inputs through ReLU and Leaky ReLU and computes the gradient of each:

import torch
from torch import nn

# Negative pre-activations, the inputs a ReLU unit turns into zeros
x = torch.tensor([-3.0, -1.0, -0.5], requires_grad=True)

for activation in (nn.ReLU(), nn.LeakyReLU(negative_slope=0.01)):
    y = activation(x)
    y.sum().backward()  # gradient that flows back toward the weights
    print(f"{activation}: output {y.detach().numpy()}, gradient {x.grad.numpy()}")
    x.grad = None

It prints ReLU(): output [0. 0. 0.], gradient [0. 0. 0.] and LeakyReLU(negative_slope=0.01): output [-0.03 -0.01 -0.005], gradient [0.01 0.01 0.01]. ReLU sends no gradient back from these inputs, so gradient descent cannot change the weights that produced them, while Leaky ReLU sends back 1% of the upstream gradient.

The Dying ReLU Problem#

A ReLU neuron dies when its weights and bias shift so far that its pre-activation is negative for every input in the training data. Its output is then always zero, its gradient is always zero, and its incoming weights never update again, so the neuron is permanently lost to the neural network. A high learning rate or a large negative bias update makes this more likely. Leaky ReLU keeps a small gradient on the negative side, so a neuron that drifts into the negative region can still be pulled back:

The dying ReLU problem is distinct from the vanishing gradient problem, in which gradients shrink gradually as they pass through many layers of saturating functions such as sigmoid or tanh.

Leaky ReLU vs. Related Activations#

• ReLU: Outputs exact zeros for negative inputs, which makes activations sparse and computation cheap but allows neurons to die. Leaky ReLU gives up exact sparsity for a gradient everywhere.
• Parametric ReLU (PReLU): Learns the negative slope during training, typically one value per channel, instead of fixing it. Kaiming He and colleagues initialized the slope at 0.25, which is also the default of PyTorch's nn.PReLU.
• SiLU and GELU: Smooth curves that dip slightly below zero for negative inputs and have no sharp corner at zero. SiLU is the default in modern YOLO models, and GELU is common in transformers. Both need an exponential per value, while Leaky ReLU needs only a comparison and a multiplication.

Where Leaky ReLU Is Used#

• DCGAN discriminators: The DCGAN architecture used Leaky ReLU with a slope of 0.2 in every discriminator layer, so the generator receives useful gradients even for images the discriminator rejects, and many later GANs kept this choice.
• Early YOLO detectors: The original YOLO used Leaky ReLU with a slope of 0.1 in every layer except the last, as did YOLOv2 and YOLOv3, and YOLOv5 used it until its January 2021 v4.0 release replaced it with SiLU.
• Graph attention networks: These graph neural networks apply Leaky ReLU with a slope of 0.2 when scoring how much each node should attend to its neighbors.
• Edge AI accelerators: Integer neural processing units that run quantized models can fold a piecewise-linear activation into the preceding convolution. AMD's Xilinx DPU does this for ReLU and Leaky ReLU but runs sigmoid, and therefore SiLU, on the host CPU.

Ultralytics YOLO model YAML files accept an activation key that replaces the default SiLU in every standard convolution block, so YOLO26 can be trained with activation: nn.LeakyReLU(0.1015625) to match the DPU's fixed slope of 26/256.