Label Smoothing

Label smoothing is a regularization technique for deep learning classification models that replaces each hard one-hot target in the training data with a softened version: most of the probability stays on the correct class, and a small share set by a smoothing factor ε is spread evenly across all classes. Training against these soft targets stops a neural network from pushing its logits toward extreme values to reach 100% confidence, which reduces overfitting and produces better-calibrated probabilities. Szegedy et al. described it in the Inception-v3 paper, using ε = 0.1 for the 1,000 ImageNet classes, and it is now a standard option when training image classification, translation, and speech models.

How Label Smoothing Works#

With K classes and smoothing factor ε, each target becomes (1 − ε) · y + ε / K, where y is the one-hot label. For three classes and ε = 0.1, the label [1, 0, 0] becomes about [0.933, 0.033, 0.033]. Against a hard target, cross-entropy keeps falling as the correct logit pulls further ahead of the others, so the loss function rewards ever-larger logit gaps and the softmax output saturates. Against a smoothed target, the loss is lowest at a finite gap of ln((1 − ε + ε / K) / (ε / K)), about 3.3 in this example, and rises again beyond it, so gradients push overconfident predictions back toward moderation. Ultralytics YOLO26 detection models train their class predictions on a related kind of soft target: the task-aligned assigner scales each positive anchor's class target by how well its predicted score and box align with the object, so most targets fall below 1.

PyTorch's CrossEntropyLoss applies the formula through its label_smoothing argument:

import torch
from torch import nn

# The correct class is index 0 of three; hard and smoothed versions of the same loss
target = torch.tensor([0])
hard_loss = nn.CrossEntropyLoss()
smooth_loss = nn.CrossEntropyLoss(label_smoothing=0.1)

# Score the correct class increasingly far above the other two
for gap in [1.0, 3.0, 6.0, 12.0]:
    logits = torch.tensor([[gap, 0.0, 0.0]])
    hard = hard_loss(logits, target).item()
    smooth = smooth_loss(logits, target).item()
    print(f"Logit gap {gap:>4}: hard loss {hard:.3f}, smoothed loss {smooth:.3f}")

The hard loss falls from 0.551 to 0.000 as the gap grows, while the smoothed loss bottoms out at 0.295 near a gap of 3 and climbs back to 0.800 at a gap of 12, so extreme confidence now costs more than moderate confidence.

Effects on Calibration and Learned Features#

Müller, Kornblith, and Hinton found that label smoothing improves model calibration, meaning predicted probabilities track how often predictions are actually right. They also showed that it pulls the penultimate-layer features of each class into tight, evenly spaced clusters. That compression erases information about which wrong classes an image resembles, and teachers trained with label smoothing transferred much less to students through knowledge distillation.

Real-World Applications#

Label smoothing helps most where labels are noisy, ambiguous, or numerous.

• ImageNet-scale classification: ImageNet contains mislabeled images and photos with several valid objects. Training recipes for convolutional networks and Vision Transformers, including DeiT, use ε = 0.1 so the model never fits a questionable label with full certainty.
• Machine translation: The original Transformer used label smoothing with ε = 0.1. The authors reported that it raised perplexity, since the model learned to be less certain, while improving accuracy and BLEU score.
• Medical image analysis: Expert annotators often disagree on borderline scans. Smoothed targets keep a diagnostic classifier from treating each disputed label as certain, and better-calibrated scores make clinical triage thresholds more reliable.

Label Smoothing vs. Related Techniques#

• Knowledge distillation: Distillation also trains on soft targets, taken from a teacher model's predictions, so they record which classes resemble each other. Label smoothing's targets give every wrong class the same small share.
• Temperature scaling: Temperature scaling calibrates a trained model after the fact by dividing its logits by one learned constant, leaving the weights untouched.
• Dropout layer: Dropout regularizes the network by zeroing random activations during training, while label smoothing regularizes through the training targets.
• CutMix: CutMix and mixup are data augmentation methods that blend two training images and mix their labels in the same proportion, so the soft target comes from the input itself. CutMix weights each label by the area its pasted patch covers, and mixup weights both pixels and labels by a single factor drawn from a Beta distribution.