Huber Loss
Huber loss combines squared and absolute error to reduce outlier influence in regression, with a tunable delta threshold and Python implementation.
Huber loss is a regression loss function that penalizes small prediction errors quadratically and large errors linearly. It gives models a smooth objective near the correct answer while reducing the influence of unusually large errors. This makes it useful when most training examples follow a consistent pattern but some contain extreme deviations.
Peter J. Huber introduced the function in his 1964 paper Robust Estimation of a Location Parameter. It remains a foundational tool for robust estimation: fitting a model whose predictions are less sensitive to outliers.
How Huber Loss Works#
A residual is the predicted value minus its target. Huber loss uses a positive threshold called delta to determine how that residual contributes to the objective:
- Small residuals: When the absolute residual is at most delta, the loss is half the squared residual.
- Large residuals: Above delta, the loss is delta x (absolute residual - 0.5 x delta).
The two branches meet with the same value and slope. During backpropagation, the derivative with respect to the prediction equals the residual inside the quadratic region; outside it, its magnitude is capped at delta. Consequently, increasingly large residuals stop producing increasingly large prediction gradients. The linear tail limits how strongly an extreme error pulls the fit away from the remaining observations.
Choosing Delta and Comparing Losses#
Mean squared error (MSE) averages squared residuals and strongly emphasizes large errors. Mean absolute error (MAE), also called L1 loss when applied per element, uses absolute residuals throughout and has a sharp corner at zero. Huber loss combines quadratic behavior near zero with linear tails, providing a continuously differentiable objective for gradient descent.
A larger delta extends the squared-error region. A smaller delta moves more residuals into the outlier-resistant linear region and lowers its slope. Delta has the same units as the residual, so target normalization changes its meaning. Converting targets from meters to centimeters requires multiplying delta by 100 to preserve the same transition relative to the data.
Delta selection is a hyperparameter-tuning problem: candidate values are compared on validation data using the intended regression metric. Huber loss suits continuous-valued prediction; cross-entropy loss and hinge loss serve classification objectives.
Applications in Regression and Computer Vision#
In linear regression, extreme observations can pull a squared-error fit away from the main data trend. A HuberRegressor fit on a generated regression dataset with four added outliers follows the main trend more closely than a squared-error fit influenced by those observations.
For object detection, Ross Girshick's Fast R-CNN uses Smooth L1 loss, equivalent to unit-delta Huber loss, for bounding-box regression. Its linear tails reduce sensitivity to outlying coordinate errors, which otherwise produce large gradients during training.
Ultralytics YOLO supports custom training criteria through a model's init_criterion() method, providing an integration point for Huber-based custom regression objectives. Such customization changes how regression errors contribute to model updates.
Huber Loss in Python#
PyTorch provides torch.nn.HuberLoss with configurable delta and reduction. After installing torch with pip install torch, the following calculation exposes each residual's contribution:
import torch
predictions = torch.tensor([0.5, 2.0, 10.0])
targets = torch.zeros_like(predictions)
# Keep individual losses to expose the linear tail.
criterion = torch.nn.HuberLoss(delta=1.0, reduction="none")
huber_losses = criterion(predictions, targets)
squared_errors = (predictions - targets).square()
print("Huber losses:", huber_losses.tolist())
print("Squared errors:", squared_errors.tolist())The output is [0.125, 1.5, 9.5] versus [0.25, 4.0, 100.0], showing the reduced penalty growth for large residuals. With reduction="mean", PyTorch averages all elementwise losses; "sum" adds them.
Pseudo-Huber loss approximates Huber's shape with a smooth transition and continuous higher-order derivatives. Smooth L1 uses a different scaling convention: in PyTorch, when beta equals delta, Huber loss equals Smooth L1 loss multiplied by delta; they are identical when both thresholds equal 1.










