Tanh (Hyperbolic Tangent)

The hyperbolic tangent, tanh(x) = (eˣ − e^(−x)) / (eˣ + e^(−x)), is an activation function that maps any real number to a value between −1 and 1 along an S-shaped curve centered on zero. It is a rescaled sigmoid, tanh(x) = 2σ(2x) − 1, so it shares sigmoid's shape but returns negative values for negative inputs and has a slope of 1 at the origin. Tanh was the standard hidden-layer activation of neural networks through the 1990s and 2000s. Today it shapes the hidden state of recurrent neural networks (RNNs), bounds the outputs of image generators and control policies, and keeps values in range inside larger models.

Formula and Properties#

Tanh is an odd function, tanh(−x) = −tanh(x), with tanh(0) = 0. Near zero it is almost linear, tanh(x) ≈ x, and beyond about ±3 it flattens toward ±1. Its derivative, 1 − tanh²(x), equals 1 at the origin, four times sigmoid's maximum of 0.25, and approaches zero as the curve saturates. The PyTorch snippet below evaluates tanh and its gradient, then compares the mean outputs of tanh and sigmoid on zero-mean inputs:

import torch

x = torch.tensor([-3.0, -1.0, 0.0, 1.0, 3.0], requires_grad=True)
y = torch.tanh(x)

# The derivative of tanh is 1 - tanh(x)^2, which peaks at 1 when x = 0
y.sum().backward()
print(f"Tanh:     {[round(v, 3) for v in y.tolist()]}")
print(f"Gradient: {[round(v, 3) for v in x.grad.tolist()]}")

# For zero-mean inputs, tanh outputs average near 0 and sigmoid outputs near 0.5
torch.manual_seed(0)
z = torch.randn(100_000)
print(f"Mean output, tanh: {torch.tanh(z).mean():.3f}, sigmoid: {torch.sigmoid(z).mean():.3f}")

It prints tanh values of [-0.995, -0.762, 0.0, 0.762, 0.995] and gradients of [0.01, 0.42, 1.0, 0.42, 0.01], so the gradient at ±3 is a hundredth of its value at zero. For 100,000 standard normal inputs, the tanh outputs average about 0.0 and the sigmoid outputs about 0.5.

Why Zero-Centered Outputs Help#

A neuron's weight gradients are proportional to its inputs. When every input is positive, as sigmoid outputs always are, all of a neuron's weight gradients share one sign in each update, so its weights can only rise together or fall together and reach their optimum by zig-zagging. Tanh outputs are positive and negative in roughly equal measure, which lets each weight move on its own and keeps the next layer's inputs near zero mean, where gradient descent converges faster. For this reason, Yann LeCun and colleagues' 1998 guide "Efficient BackProp" recommended symmetric functions over the logistic sigmoid for hidden layers, specifically the scaled form 1.7159 · tanh(2x/3). Tanh still saturates, and stacking many tanh layers shrinks gradients during backpropagation, the vanishing gradient problem.

Where Tanh Is Used#

• Recurrent networks: A simple RNN updates its state as hₜ = tanh(Wxₜ + Uhₜ₋₁ + b), and tanh is the default nonlinearity of PyTorch's nn.RNN. In LSTM and GRU cells, tanh produces the candidate content and, in the LSTM, squashes the cell state before output, while sigmoid gates decide how much of it to use. The bounded range keeps a state that is fed back step after step from growing without limit.
• GAN generators: DCGAN ends its generator with tanh so that generated pixels fall between −1 and 1, the range its training images are scaled to, a choice many later generative adversarial networks (GANs) follow.
• Bounded actions: Policies for continuous control in reinforcement learning pass their outputs through tanh to keep every command within actuator limits. Soft Actor-Critic squashes samples from a Gaussian policy this way.
• Logit soft-capping: Google's Gemma 2 language models compute cap · tanh(logits / cap), with a cap of 50 for attention logits and 30 for the final logits. Small logits pass almost unchanged, and large ones can never exceed the cap.
• GELU approximation: 0.5x(1 + tanh(√(2/π)(x + 0.044715x³))) approximates GELU closely, and the original GPT-2 and BERT code computes GELU this way.
• Normalizing flows: RealNVP-style flows often end each scale network with tanh, so every coupling layer scales its inputs by a factor between e⁻¹ and e, which keeps training numerically stable. YOLO26 pose estimation models train with a residual log-likelihood estimation loss whose six-layer RealNVP flow, with tanh closing every scale network, models how far predicted keypoints fall from the ground truth.

Tanh vs. Related Functions#

• Sigmoid: Tanh is sigmoid stretched to the range −1 to 1 and compressed horizontally by a factor of two, which raises the maximum slope from 0.25 to 1. Sigmoid's 0-to-1 range makes it the function of choice for probabilities and gates.
• Hardtanh: A piecewise-linear version that clips inputs to [−1, 1]. It is cheaper to compute, and its gradient is exactly 0 outside that range.
• ReLU: max(0, x) never saturates for positive inputs and costs one comparison, so ReLU and smooth successors such as SiLU replaced tanh in the hidden layers of deep convolutional neural networks (CNNs) and transformers. In the AlexNet paper, a four-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh units.