Loss Function

A loss function measures how far a model's predictions are from the ground-truth labels in its training data and condenses that error into a single number, where lower is better. Training a neural network means adjusting its model weights to make this number as small as possible. "Cost function" and "objective function" are common synonyms, although some texts reserve "loss" for the error on a single example and "cost" for the average over a batch or dataset.

How Loss Drives Training#

Training repeats the same cycle for every batch of data. The model makes predictions, the loss function scores them against the labels, and backpropagation computes the gradient of the loss with respect to every weight. An optimizer such as stochastic gradient descent (SGD) or the Adam optimizer then moves each weight a small step against its gradient, with the step size set by the learning rate. Because this process relies on gradients, a loss function must be differentiable almost everywhere. Plotting training and validation loss after each epoch is the standard way to monitor a run: a validation loss that rises while training loss keeps falling is the classic sign of overfitting.

Common Loss Functions#

The right loss depends on what the model predicts, and frameworks such as PyTorch ship most of the standard choices:

• Mean squared error (MSE): The average squared difference between predicted and true values, used for regression. Squaring makes large errors dominate, so outliers have a strong pull.
• Mean absolute error (L1) and Huber loss: Regression losses that grow linearly with large errors, which makes them less sensitive to outliers. Huber loss is quadratic for small errors and linear beyond a threshold.
• Cross-entropy: The standard loss for image classification, computed on softmax probabilities over mutually exclusive classes.
• Binary cross-entropy (BCE): Scores each class as an independent yes-or-no prediction through a sigmoid, which suits multi-label problems and detector class scores.
• Focal loss: A cross-entropy variant that down-weights easy examples so rare, hard ones drive learning under heavy class imbalance.
• IoU-based losses: Box regression losses built on Intersection over Union (IoU), with variants such as DIoU and CIoU that add penalties for center distance and, in CIoU, aspect ratio.
• Dice loss: An overlap loss for image segmentation that stays informative when the foreground covers only a small fraction of the pixels.
• Contrastive losses: Losses for contrastive learning that pull matching pairs together in embedding space and push mismatched pairs apart.

Cross-entropy uses the negative log of the probability assigned to the true class, so a confident mistake costs far more than a hesitant one, and a confident correct answer costs almost nothing. PyTorch's CrossEntropyLoss shows the effect on two predictions with the same confidence:

import torch
from torch import nn

# CrossEntropyLoss applies softmax to raw scores (logits) internally
loss_fn = nn.CrossEntropyLoss()

# Logits for 3 classes; the true class is index 0
target = torch.tensor([0])
confident_right = torch.tensor([[4.0, 0.5, -1.0]])
confident_wrong = torch.tensor([[-1.0, 4.0, 0.5]])

print(f"Confident and right: {loss_fn(confident_right, target).item():.4f}")
print(f"Confident and wrong: {loss_fn(confident_wrong, target).item():.4f}")

The confident correct prediction costs 0.0363, while the same confidence placed on the wrong class costs 5.0363, about 139 times more.

Composite Losses in Object Detection#

Object detection models predict where each object is and what it is, so their training loss is a weighted sum of several terms. YOLO26 combines a CIoU box loss, a BCE classification loss, and an L1 loss on normalized box distances that replaces the Distribution Focal Loss of earlier YOLO models, weighted by the box, cls, and dfl training arguments (defaults 7.5, 0.5, and 1.5). Its Progressive Loss also shifts weight from the one-to-many head toward the one-to-one head used for NMS-free inference as training advances.

Loss Function vs. Related Terms#

• Evaluation metrics: Metrics such as accuracy and mean average precision (mAP) report performance in terms people care about, but most change in discrete jumps and cannot supply gradients. Training minimizes a differentiable loss as a proxy for these metrics.
• Regularization: Penalty terms added to the loss to discourage complexity. L2 regularization adds a multiple of the sum of squared weights, which pushes the optimizer toward smaller weights that generalize better.
• Reward function: The objective in reinforcement learning, where an agent maximizes cumulative reward instead of minimizing a loss, so a reward plays the role of a negative loss.