Sigmoid

The sigmoid function, σ(x) = 1 / (1 + e^(−x)), maps any real number to a value between 0 and 1 along an S-shaped curve: large negative inputs give outputs near 0, large positive inputs give outputs near 1, and an input of 0 gives exactly 0.5. In machine learning (ML), "sigmoid" almost always means this logistic function, the same curve that turns a linear score into a probability in logistic regression. Inside a neural network (NN), sigmoid plays three roles: it was the original hidden-layer activation function, it turns a final score into a probability for binary and multi-label outputs, and it acts as a soft on-off switch in the gates of recurrent cells and other gated layers. Hidden layers have since moved to functions such as ReLU, so most sigmoids in current models sit at outputs and in gates.

Formula and Properties#

Sigmoid is symmetric around its midpoint, σ(−x) = 1 − σ(x), and its inverse is the logit function, log(p / (1 − p)), which converts a probability back into log-odds. That is why a network's raw output before a sigmoid is called a logit. The derivative has a compact form, σ′(x) = σ(x)(1 − σ(x)), which peaks at 0.25 when x = 0 and falls toward zero as the curve flattens at either end, a behavior called saturation. The PyTorch snippet below computes outputs and gradients at five points:

import torch

x = torch.tensor([-6.0, -2.0, 0.0, 2.0, 6.0], requires_grad=True)
y = torch.sigmoid(x)

# Backpropagate to get the derivative sigmoid(x) * (1 - sigmoid(x)) at each input
y.sum().backward()

print(f"Sigmoid:  {[round(v, 4) for v in y.tolist()]}")
print(f"Gradient: {[round(v, 4) for v in x.grad.tolist()]}")

It prints sigmoid values of [0.0025, 0.1192, 0.5, 0.8808, 0.9975] and gradients of [0.0025, 0.105, 0.25, 0.105, 0.0025]. At ±6 the output is within 0.0025 of its limit and the gradient is 100 times smaller than at zero, so an input in that range barely changes the weights that produced it.

Sigmoid in Hidden Layers#

Early multilayer networks used sigmoid in every hidden layer, which caused two problems. Backpropagation multiplies one activation derivative per layer, and because sigmoid's derivative never exceeds 0.25, across ten sigmoid layers those derivatives alone scale the gradient by at most 0.25¹⁰, less than one millionth, the vanishing gradient problem. Sigmoid outputs are also always positive, so all the weight gradients of a neuron that receives them share one sign in each update, and the weights can only zig-zag toward their optimum. Tanh centers its outputs on zero, and Xavier Glorot and Yoshua Bengio showed in 2010 that sigmoid's positive mean drives the top hidden layer of a randomly initialized deep network into saturation. ReLU removed both problems and became the default for hidden layers, and smooth successors such as SiLU, x · σ(x), still use sigmoid inside them as a gate.

Sigmoid as an Output Layer#

For binary classification, a single output z becomes p = σ(z), the predicted probability of the positive class, and a threshold of 0.5 on p is the same as a threshold of 0 on z. Training pairs sigmoid with binary cross-entropy, whose gradient with respect to the logit is simply p − y, the predicted probability minus the label. Frameworks fuse the two for numerical stability: PyTorch's BCEWithLogitsLoss takes raw logits and uses the log-sum-exp trick internally, which avoids taking the log of a probability that has rounded to exactly 0 or 1.

When labels can co-occur, a model uses one sigmoid per label, a setup called multi-label classification. Softmax instead makes the classes compete for a single unit of probability, which suits labels that exclude each other:

import torch

# Logits from a classifier for the labels "person", "backpack", and "car"
logits = torch.tensor([2.0, 1.5, -3.0])

# Sigmoid scores each label on its own; softmax makes the labels compete for one total
sigmoid_probs = torch.sigmoid(logits)
softmax_probs = torch.softmax(logits, dim=0)

print(f"Sigmoid: {[round(p, 3) for p in sigmoid_probs.tolist()]}, sum {sigmoid_probs.sum():.3f}")
print(f"Softmax: {[round(p, 3) for p in softmax_probs.tolist()]}, sum {softmax_probs.sum():.3f}")

The sigmoid scores [0.881, 0.818, 0.047] sum to 1.746, so "person" and "backpack" both clear 0.5, while softmax splits the same logits into [0.62, 0.376, 0.004] and only "person" stays above 0.5. YOLO26 detection models score each class with an independent sigmoid trained with BCEWithLogitsLoss, so a box's class scores need not sum to 1, and YOLO26 pose models apply a sigmoid to each keypoint's visibility logit to output the probability that the keypoint is visible.

Sigmoid as a Gate#

Because its output lies between 0 and 1, sigmoid can scale another signal element by element, anywhere from fully blocked to fully passed, while staying differentiable:

• LSTM and GRU cells: Sigmoid gates decide how much of the old memory to keep, how much new content to write, and how much of the state to expose at each time step, while tanh shapes the content itself.
• SwiGLU: The feed-forward layers of many large language models (LLMs) multiply one learned projection by a SiLU-activated second projection, a gate built on sigmoid.
• SigLIP: This image-text model scores every image-caption pair in a batch with its own sigmoid, a yes-or-no match probability, so its loss needs no normalization across the batch.

Sigmoid vs. Related Functions#

• Logistic function: A synonym for sigmoid in ML. In mathematics, "sigmoid" also covers other S-shaped curves, including tanh and the error function.
• Softmax: Normalizes a whole vector into probabilities that sum to 1. For two classes, softmax over the logits (z₁, z₂) gives the first class σ(z₁ − z₂), so a two-output softmax and a one-output sigmoid describe the same model.
• Tanh: A rescaled sigmoid, tanh(x) = 2σ(2x) − 1, with outputs from −1 to 1 centered on zero and a maximum derivative of 1.
• Hard sigmoid: A piecewise-linear approximation that avoids the exponential on mobile hardware. MobileNetV3 uses ReLU6(x + 3) / 6, which is 0 below x = −3, 1 above x = 3, and a straight line in between.