Uncertainty Quantification

Uncertainty quantification (UQ) is the process of estimating, evaluating, and communicating how uncertain an AI or machine learning model is about its predictions. Instead of returning only a label, value, or bounding box, a UQ workflow provides additional information such as a probability distribution, prediction interval, confidence score, or prediction set. This helps users understand not just what a model predicts, but also how much trust to place in that prediction.

Link to this sectionWhy Uncertainty Quantification Matters#

Models learn patterns from limited data, so their outputs are never perfectly certain. Inputs may be noisy, labels may be ambiguous, and production data may differ from the training distribution. UQ makes these limitations measurable and supports safer decisions, human review, and better data collection.

This idea extends the broader goal of characterizing model quality described by NIST’s virtual measurement program. In AI systems, uncertainty estimates can help teams:

• Reject or review predictions when uncertainty exceeds an acceptable level.
• Compare alternative predictions rather than relying on a single answer.
• Identify unfamiliar inputs and possible data drift.
• Prioritize uncertain samples for labeling through active learning.
• Propagate uncertainty into downstream calculations and decisions, as illustrated by the Department of Energy’s overview of uncertainty in computational models.

UQ is especially important when prediction errors have unequal consequences. Missing a pedestrian is more serious than incorrectly detecting a roadside sign, so the acceptable uncertainty threshold depends on the application.

Link to this sectionAleatoric and Epistemic Uncertainty#

Two categories explain where predictive uncertainty comes from:

• Aleatoric uncertainty, also called aleatory uncertainty, comes from inherent randomness or ambiguity in the data. Motion blur, sensor noise, partial occlusion, and inconsistent labels can make an image difficult to interpret even for a well-trained model. More training data may help estimate this uncertainty, but it cannot completely remove the underlying ambiguity.
• Epistemic uncertainty comes from incomplete model knowledge. It often appears when training data is insufficient, unrepresentative, or missing important situations. It can frequently be reduced by collecting relevant examples, improving annotations, or changing the model.

UQ combines information about these sources into useful outputs. Probabilistic modeling tools such as TensorFlow Probability can represent predictions as distributions, while ensemble methods estimate uncertainty by comparing multiple models or training runs. Large disagreement generally suggests greater epistemic uncertainty.

Uncertainty is related to, but broader than, a confidence score. Confidence describes the strength of one prediction, whereas UQ evaluates whether that score is reliable and what sources of uncertainty affect it. Similarly, conformal prediction is a specific technique for producing prediction sets or intervals with target coverage under defined assumptions.

Link to this sectionMethods and Evaluation#

Common UQ approaches include probabilistic outputs, ensembles, repeated sampling, Bayesian modeling, and prediction intervals. Bootstrap confidence intervals estimate variability by repeatedly resampling observed data. For classification, calibration checks whether predictions assigned approximately 80% probability are correct about 80% of the time.

A model can be accurate but poorly calibrated, so teams should evaluate both task performance and uncertainty quality. Probability calibration tools use reliability diagrams and calibration techniques to compare predicted probabilities with observed outcomes. Google’s Rules of Machine Learning also emphasizes interpretable, monitored probabilistic predictions.

Useful evaluation criteria include coverage, interval width, calibration error, and performance on difficult data slices. For computer vision, Ultralytics validation mode and the YOLO performance metrics guide help examine precision, recall, confusion matrices, and confidence behavior across classes and thresholds.

Link to this sectionReal-World Applications#

• Medical image analysis: A diagnostic model can flag a scan for specialist review when several plausible findings have similar scores or when the image differs from its training data. Without this escalation, an overconfident incorrect prediction could delay treatment.
• Manufacturing visual inspection: A defect detector may encounter glare, new materials, or previously unseen damage. Uncertainty-aware routing can send ambiguous products to manual inspection instead of automatically accepting or rejecting them, reducing both escaped defects and unnecessary waste.

These policies connect model uncertainty to operational risk. The NIST AI Risk Management Framework Core recommends measuring uncertainty, documenting generalization limits, and monitoring systems throughout deployment.

Link to this sectionUncertainty in Ultralytics YOLO Workflows#

Ultralytics YOLO26 exposes detection confidence scores through its documented Predict mode:

from ultralytics import YOLO

# Load an official detection model
model = YOLO("yolo26n.pt")

# Run inference
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Inspect each detection's confidence
for box in result.boxes:
    class_id = int(box.cls.item())
    class_name = result.names[class_id]
    confidence = box.conf.item()
    print(f"{class_name}: {confidence:.3f}")

This workflow exposes a useful uncertainty-related signal, but confidence alone is not complete UQ and should not automatically be interpreted as a calibrated probability. Validate scores on representative held-out data, choose thresholds based on error costs, and monitor confidence distributions after deployment.

Teams can use Ultralytics Platform to annotate datasets, train and deploy models, and monitor endpoints. Combined with ongoing model monitoring and maintenance, uncertainty signals can reveal unfamiliar conditions, guide human review, and identify when new data or retraining is needed.