Uncertainty Quantification

Uncertainty quantification (UQ) is the practice of estimating how uncertain a machine learning model is about each prediction, and of checking that those estimates are reliable. Along with a label, value, or bounding box, a model with UQ returns a probability distribution, a prediction interval, a set of plausible labels, or a variance. Systems use these estimates to decide when to act automatically, when to defer to a person, and which new data to collect. A confidence score is one such estimate; UQ also asks whether that score is calibrated, meaning that predictions made with 80% confidence are right about 80% of the time.

Aleatoric and Epistemic Uncertainty#

Predictive uncertainty has two sources, a split that Alex Kendall and Yarin Gal brought into deep learning for vision in their 2017 paper What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?:

• Aleatoric uncertainty: Noise in the data itself, such as motion blur, sensor noise, occlusion, or labels that annotators disagree on. More training data does not remove it, because the input does not contain the missing information. A model can learn to predict it as an extra output, such as a variance for each depth estimate.
• Epistemic uncertainty: The model's lack of knowledge, which is high for inputs unlike anything in its training data, such as a new camera, season, or object type. It shrinks as relevant data is added, so it signals data drift and out-of-distribution inputs.

The distinction decides what to do with an uncertain prediction:

Methods for Estimating Uncertainty#

• Predicted probabilities: The softmax output of a classifier is the cheapest estimate, but a 2017 study by Chuan Guo and colleagues found modern deep networks systematically overconfident. Temperature scaling on held-out data corrects much of the gap.
• Monte Carlo dropout: Keeps dropout active at inference and runs the input several times, treating the spread of outputs as approximate Bayesian uncertainty, as Yarin Gal and Zoubin Ghahramani showed in 2016.
• Deep ensembles: Train several copies of a network from different random initializations and measure their disagreement, a method Balaji Lakshminarayanan and colleagues at DeepMind introduced in 2017. Ensembles held up best under dataset shift in a large 2019 Google comparison of UQ methods.
• Conformal prediction: Wraps any model to output prediction sets or intervals that contain the true answer at a chosen rate, such as 95%.
• Semantic entropy: Samples several answers from a large language model (LLM) and measures how many distinct meanings they express, a signal for detecting hallucinations.

A deep ensemble on a one-dimensional regression task shows epistemic uncertainty directly:

import numpy as np
from sklearn.neural_network import MLPRegressor

# Noisy samples of sin(x), observed only between -3 and 3
rng = np.random.default_rng(0)
x = rng.uniform(-3, 3, (200, 1))
y = np.sin(x).ravel() + rng.normal(0, 0.1, 200)

# Deep ensemble: the same network trained from five random initializations
ensemble = [MLPRegressor(hidden_layer_sizes=(32, 32), max_iter=3000, random_state=seed).fit(x, y) for seed in range(5)]

# Members agree where data exists and diverge where it does not
for x_new in (1.0, 8.0):
    preds = np.array([m.predict([[x_new]])[0] for m in ensemble])
    print(f"x={x_new}: mean {preds.mean():+.2f}, std {preds.std():.2f}")

At x = 1, inside the training range, the five networks predict a mean of 0.86 against a true value of 0.84, with a standard deviation of 0.01. At x = 8, far outside it, they predict -1.17 for a true value of 0.99, and the standard deviation jumps to 0.59, so the ensemble flags the very prediction that is wrong.

Evaluating Uncertainty Estimates#

Calibration is checked with a reliability diagram, which bins predictions by confidence and plots each bin's actual accuracy, and summarized by expected calibration error or the Brier score. Prediction intervals are judged by coverage, the share of true values that fall inside them, and by width, since an interval wide enough to cover everything carries no information. Out-of-distribution detection is scored with the area under the ROC curve for separating familiar from unfamiliar inputs, and benchmarks now report calibration next to accuracy: Humanity's Last Exam found launch-era LLMs with calibration errors between 73% and 89%.

Applications#

• Medical image analysis: Diagnostic models in healthcare refer scans with high uncertainty to a radiologist, so automation covers the clear cases and people review the ambiguous ones.
• Autonomous vehicles: Perception systems widen safety margins or hand control back when depth or detection uncertainty rises, as in fog or glare.
• Active learning: Uncertainty decides which unlabeled images are worth annotating, because the samples a model is least sure about teach it the most.

Active learning is where detection pipelines use uncertainty most directly. The Collect Uncertain Detections template in Ultralytics Platform Agents runs a dataset's images through a deployed YOLO26 model and copies every image with at least one detection whose confidence falls in a chosen range, such as 0.25 to 0.5, into a review dataset, unlabeled, ready for annotation and retraining.