Sigmoid

The Sigmoid function is a popular activation function used in machine learning (ML) and deep learning (DL). It is a mathematical function that produces a characteristic "S"-shaped, or sigmoidal, curve. Its primary purpose is to take any real-valued number and "squash" it into a range between 0 and 1. This output is often interpreted as a probability, making Sigmoid especially useful in models where the goal is to predict the likelihood of an outcome. By introducing non-linearity into a neural network (NN), it enables the model to learn complex patterns from data that would otherwise be impossible with simple linear transformations.

Role and Applications

The Sigmoid function's ability to map inputs to a probability-like output makes it a cornerstone for certain types of tasks. While it has become less common in the hidden layers of modern deep neural networks, it remains a standard choice for the output layer in specific scenarios.

Key Applications

• Binary Classification: In binary classification problems, the goal is to categorize an input into one of two classes (e.g., spam or not spam, disease present or absent). A Sigmoid function at the output layer provides a single value between 0 and 1, representing the probability that the input belongs to the positive class. For instance, a medical image analysis model might use Sigmoid to output a probability of 0.9, indicating a 90% chance that a tumor is malignant.
• Multi-Label Classification: Unlike multi-class classification where an input belongs to only one class, multi-label tasks allow an input to be associated with multiple labels simultaneously. For example, an object detection model like Ultralytics YOLO might analyze an image and identify a "car," "pedestrian," and "traffic light" all at once. In this case, a Sigmoid function is applied to each output neuron independently, giving the probability for each possible label. You can learn more about the evolution of object detection.
• Gating Mechanisms in RNNs: Sigmoid functions are a core component in the gating mechanisms of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). These gates use Sigmoid to control the flow of information, deciding what data to keep or discard at each step. This mechanism is crucial for learning long-term dependencies in sequential data, as explained in this detailed blog post on understanding LSTMs.

Comparison With Other Activation Functions

It's important to distinguish the Sigmoid function from other activation functions to understand when to use it.

• Softmax: The Softmax function is typically used for multi-class classification problems, where each input belongs to exactly one of several possible classes. Unlike Sigmoid, which calculates independent probabilities for each output, Softmax calculates a probability distribution across all classes that sums to 1. For example, a model classifying handwritten digits from the MNIST dataset would use Softmax to assign a single probability to each digit from 0 to 9.
• ReLU (Rectified Linear Unit): ReLU has become the de-facto standard for hidden layers in deep networks. It is computationally more efficient and helps mitigate the vanishing gradient problem—a significant issue with Sigmoid where the gradients become extremely small during backpropagation, slowing down or halting the learning process. You can read more about the challenges of gradients in this DeepLearning.AI article.
• SiLU (Sigmoid Linear Unit): Also known as Swish, SiLU is a more modern activation function derived from Sigmoid. It often performs better than ReLU in deeper models, including advanced computer vision architectures. Ultralytics models often leverage advanced activation functions to achieve a better balance of speed and accuracy.

Modern Usage And Availability

While less common in hidden layers today, Sigmoid remains a standard choice for output layers in binary and multi-label classification tasks. It also forms a core component in gating mechanisms within complex architectures that handle sequential data.

Sigmoid is readily available in all major deep learning frameworks, including PyTorch (as torch.sigmoid) and TensorFlow (as tf.keras.activations.sigmoid). Platforms like Ultralytics HUB support models utilizing various activation functions, allowing users to train and deploy sophisticated computer vision solutions.