Cross-Entropy Loss

Cross-entropy loss measures how closely a model's predicted class probabilities match its target labels. It assigns a small penalty when the correct class receives high probability and a large penalty when that probability is low. As a loss function for supervised learning, it supplies an objective to minimize during training.

Claude Shannon's 1948 A Mathematical Theory of Communication established its entropy foundation. D. F. Kerridge introduced the cross-entropy measure as "inaccuracy" in his 1961 paper Inaccuracy and Inference. In classification, log loss is a synonym; logistic regression and neural classifiers both use it.

How Cross-Entropy Loss Works#

For one example with one correct class, unweighted cross-entropy is loss = -ln(p_correct). A correct-class probability of 0.9 gives a loss of about 0.105; 0.1 gives about 2.303. Natural logarithms express the result in units called nats. Training usually averages these penalties across a batch.

More generally, cross-entropy multiplies each target probability by the negative logarithm of its predicted probability and sums across classes. A one-hot target places 1 on the correct class and 0 elsewhere, leaving only the correct-class term. For unweighted, one-hot targets, minimizing cross-entropy is equivalent to maximizing the likelihood of observed labels. Backpropagation computes gradients, and gradient descent updates the model's parameters.

Binary, Categorical, and Sparse Cross-Entropy#

The appropriate form depends on the labels:

• Binary cross-entropy with sigmoid: For a 0-or-1 target y and predicted probability p of class 1, the loss is -[y * ln(p) + (1 - y) * ln(1 - p)]. Multi-label classification applies it independently to labels that can occur together.
• Categorical cross-entropy with softmax: This scores mutually exclusive classes whose probabilities sum to 1. Softmax converts raw model scores, called logits, into that distribution.
• Sparse categorical cross-entropy: This uses an integer class index instead of a one-hot vector. It calculates the same loss; "sparse" describes the target encoding.

Soft targets distribute probability across classes. They appear in label smoothing and knowledge distillation, where a student learns from a teacher's probability distribution.

Computing Cross-Entropy in PyTorch#

PyTorch accepts raw logits in CrossEntropyLoss. For class-index targets, its calculation is equivalent to log-softmax followed by negative log-likelihood. Applying softmax beforehand makes the function treat probabilities as logits, changing the loss and gradients. Integer targets use torch.long; probability targets use floating-point values and match the logits' shape.

import torch
from torch import nn

# Both predictions select class 0; the second is more confident.
logits = torch.tensor([[2.0, 0.0, 0.0], [4.0, 0.0, 0.0]])
targets = torch.tensor([0, 0], dtype=torch.long)
loss_fn = nn.CrossEntropyLoss(reduction="none")

losses = loss_fn(logits, targets)
print(f"Lower confidence loss: {losses[0].item():.4f}")
print(f"Higher confidence loss: {losses[1].item():.4f}")

The losses decrease from 0.2395 to 0.0360 while both predictions remain correct. Accuracy records class decisions; cross-entropy also responds to probability assignments. For binary or multi-label logits, BCEWithLogitsLoss combines sigmoid and binary cross-entropy.

Applications in Computer Vision#

Single-label image classification uses cross-entropy to learn one class per image. Ultralytics YOLO26 classification training uses mean cross-entropy over class logits to update the classifier toward the labeled category.

In healthcare image segmentation, the original U-Net trained on biomedical microscopy images with pixelwise softmax and weighted cross-entropy. Its weight map increased penalties at boundaries between touching cells, helping separate neighboring cells. This applies classification at every pixel in semantic segmentation.

Interpreting Loss Values#

Discrete cross-entropy with valid target distributions and nonnegative weights is nonnegative. For an unweighted hard target, it reaches zero when the correct class receives probability 1 and grows without bound as that probability approaches zero. Negative values indicate an input or implementation error, such as negative target entries or weights.

Entropy measures uncertainty within one distribution; cross-entropy compares targets with predictions. Cross-entropy equals target entropy plus Kullback–Leibler divergence, the additional mismatch penalty. Soft targets therefore have a positive minimum when their entropy is positive.

Class weighting increases selected classes' contributions. Focal loss reduces easy examples' contributions to address foreground–background imbalance in object detection. For a uniform prediction, unweighted log loss equals ln(number_of_classes): approximately 0.693 for two classes and 2.303 for ten.