Confidence Interval
A confidence interval quantifies sampling uncertainty in model metrics such as accuracy and mAP. A 95% interval describes long-run coverage.
A confidence interval (CI) is a range calculated from a sample using a method designed to capture an unknown population quantity at a specified rate. In machine learning, a test score estimates how a fixed model performs on the kinds of inputs the test set represents. The interval shows how precisely that performance is estimated from a finite sample.
What a 95% Confidence Interval Means#
If you repeatedly drew comparable samples of test data and calculated an interval the same way each time, a procedure with 95% coverage would capture the true metric in about 95 out of 100 intervals over the long run. The metric is fixed; the interval bounds change from sample to sample. Jerzy Neyman introduced this frequentist framework in his 1934 paper on sampling.
A higher confidence level usually produces a wider interval from the same data. More independent observations generally make the interval narrower at a fixed confidence level. Width measures sampling precision: an interval can be narrow even when dataset bias makes the test sample unrepresentative of deployment conditions.
How to Calculate a Confidence Interval#
For a sample mean, a common formula is estimate ± margin of error, where the margin of error is the critical value multiplied by the standard error. The standard error measures how much the estimate varies across repeated samples. For independent observations from the same population, the estimated standard error of the mean is the sample standard deviation divided by the square root of the sample size. A two-sided 95% normal interval uses a critical value of 1.96, leaving 2.5% in each tail. A one-sample t interval uses a Student's t critical value with sample size minus one degrees of freedom. Its coverage is exact for independent observations from the same normal population.
The method depends on the metric and how the test data were sampled:
- Classification accuracy: With independent images and a common probability of a correct classification, a Wilson score interval uses the count of correct predictions and the test-set size to estimate that probability. Both bounds stay between 0% and 100%.
- Mean average precision (mAP): Bootstrap the full metric by resampling test units with replacement and recomputing mAP for each draw. A 95% percentile interval uses the 2.5th and 97.5th percentiles of those scores. In an image-level bootstrap, each image carries its predictions and annotations together.
Worked Example: Handwritten Digit Accuracy#
The scikit-learn digit-classification example fits a support vector machine on the first half of the bundled digit images and evaluates it on the second half. The code uses the same split and calculates a Wilson interval with SciPy's binomial proportion API.
from scipy.stats import binomtest
from sklearn.datasets import load_digits
from sklearn.svm import SVC
X, y = load_digits(return_X_y=True)
split = len(y) // 2
model = SVC(gamma=0.001)
model.fit(X[:split], y[:split])
predictions = model.predict(X[split:])
correct = int((predictions == y[split:]).sum())
accuracy = binomtest(correct, len(y[split:]))
interval = accuracy.proportion_ci(confidence_level=0.95, method="wilson")
print(f"Accuracy: {accuracy.statistic:.1%} (95% CI: {interval.low:.1%}–{interval.high:.1%})")The classifier gets 871 of 899 images right: 96.9% accuracy with a 95% Wilson interval of 95.5%–97.8% under the independent-image model. The bundled images come from the UCI handwritten digits test set, which was drawn from 13 writers. Images from the same writer can share handwriting traits, so estimating accuracy for new writers calls for accounting for that grouping.
Confidence Intervals, Scores, and Predictions#
A confidence score is a model output for one prediction. In Ultralytics YOLO prediction mode, conf sets the minimum score a detection needs to be retained. A confidence interval for recall at that threshold quantifies uncertainty in the recall estimate from the test sample.
A prediction interval targets a future observation. For example, conformal prediction has been used to construct per-pixel prediction intervals on KITTI for depth estimates in new frames.
Comparing Computer Vision Models#
An interval for a fixed model measures test-set sampling uncertainty. Comparing training pipelines also involves variation from factors such as random initialization and hyperparameter tuning, which can be measured through repeated training runs.
When two object detectors run on the same test set, a paired bootstrap draws the same units for both models and calculates the difference in mAP on each draw. Independently sampled still images can be drawn one at a time. When independently sampled videos contain correlated frames, drawing whole videos preserves the dependence between frames. Treating neighboring frames as independent can make the interval too narrow. Because pairing preserves the association between results on the same inputs, the interval for the mAP difference can exclude zero even when the two individual confidence intervals overlap.










