License YOLO
Back to Ultralytics Glossary

Confidence Interval

A confidence interval quantifies sampling uncertainty in model metrics such as accuracy and mAP. A 95% interval describes long-run coverage.

A confidence interval (CI) is a range calculated from a sample using a method designed to capture an unknown population quantity at a specified rate. In machine learning, a test score estimates how a fixed model performs on the kinds of inputs the test set represents. The interval shows how precisely that performance is estimated from a finite sample.

What a 95% Confidence Interval Means#

If you repeatedly drew comparable samples of test data and calculated an interval the same way each time, a procedure with 95% coverage would capture the true metric in about 95 out of 100 intervals over the long run. The metric is fixed; the interval bounds change from sample to sample. Jerzy Neyman introduced this frequentist framework in his 1934 paper on sampling.

A higher confidence level usually produces a wider interval from the same data. More independent observations generally make the interval narrower at a fixed confidence level. Width measures sampling precision: an interval can be narrow even when dataset bias makes the test sample unrepresentative of deployment conditions.

How to Calculate a Confidence Interval#

For a sample mean, a common formula is estimate ± margin of error, where the margin of error is the critical value multiplied by the standard error. The standard error measures how much the estimate varies across repeated samples. For independent observations from the same population, the estimated standard error of the mean is the sample standard deviation divided by the square root of the sample size. A two-sided 95% normal interval uses a critical value of 1.96, leaving 2.5% in each tail. A one-sample t interval uses a Student's t critical value with sample size minus one degrees of freedom. Its coverage is exact for independent observations from the same normal population.

The method depends on the metric and how the test data were sampled:

  • Classification accuracy: With independent images and a common probability of a correct classification, a Wilson score interval uses the count of correct predictions and the test-set size to estimate that probability. Both bounds stay between 0% and 100%.
  • Mean average precision (mAP): Bootstrap the full metric by resampling test units with replacement and recomputing mAP for each draw. A 95% percentile interval uses the 2.5th and 97.5th percentiles of those scores. In an image-level bootstrap, each image carries its predictions and annotations together.

Worked Example: Handwritten Digit Accuracy#

The scikit-learn digit-classification example fits a support vector machine on the first half of the bundled digit images and evaluates it on the second half. The code uses the same split and calculates a Wilson interval with SciPy's binomial proportion API.

from scipy.stats import binomtest
from sklearn.datasets import load_digits
from sklearn.svm import SVC

X, y = load_digits(return_X_y=True)
split = len(y) // 2
model = SVC(gamma=0.001)
model.fit(X[:split], y[:split])
predictions = model.predict(X[split:])

correct = int((predictions == y[split:]).sum())
accuracy = binomtest(correct, len(y[split:]))
interval = accuracy.proportion_ci(confidence_level=0.95, method="wilson")
print(f"Accuracy: {accuracy.statistic:.1%} (95% CI: {interval.low:.1%}–{interval.high:.1%})")

The classifier gets 871 of 899 images right: 96.9% accuracy with a 95% Wilson interval of 95.5%–97.8% under the independent-image model. The bundled images come from the UCI handwritten digits test set, which was drawn from 13 writers. Images from the same writer can share handwriting traits, so estimating accuracy for new writers calls for accounting for that grouping.

Confidence Intervals, Scores, and Predictions#

A confidence score is a model output for one prediction. In Ultralytics YOLO prediction mode, conf sets the minimum score a detection needs to be retained. A confidence interval for recall at that threshold quantifies uncertainty in the recall estimate from the test sample.

A prediction interval targets a future observation. For example, conformal prediction has been used to construct per-pixel prediction intervals on KITTI for depth estimates in new frames.

Comparing Computer Vision Models#

An interval for a fixed model measures test-set sampling uncertainty. Comparing training pipelines also involves variation from factors such as random initialization and hyperparameter tuning, which can be measured through repeated training runs.

When two object detectors run on the same test set, a paired bootstrap draws the same units for both models and calculates the difference in mAP on each draw. Independently sampled still images can be drawn one at a time. When independently sampled videos contain correlated frames, drawing whole videos preserves the dependence between frames. Treating neighboring frames as independent can make the interval too narrow. Because pairing preserves the association between results on the same inputs, the interval for the mAP difference can exclude zero even when the two individual confidence intervals overlap.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning