Conformal Prediction

Conformal prediction is a framework for uncertainty quantification that turns the scores of any trained model into prediction sets or intervals guaranteed to contain the true answer at a rate the user chooses, such as 95%. A conformal image classifier returns every label that is still plausible, one label for a clear image and several for an ambiguous one, and a conformal regressor returns an interval. The guarantee holds for any model and any data distribution, provided the calibration data and new data are exchangeable, which independent samples from the same source are. Vladimir Vovk, Alexander Gammerman, and Glenn Shafer developed the method from the late 1990s and set it out in their 2005 book Algorithmic Learning in a Random World, and Anastasios Angelopoulos and Stephen Bates's 2021 tutorial A Gentle Introduction to Conformal Prediction covers the modern methods used in machine learning.

How Split Conformal Prediction Works#

The original full conformal method refits the model for every candidate answer, which is too slow for deep learning. Split, or inductive, conformal prediction, introduced by Harris Papadopoulos and colleagues in 2002, needs only one trained model and a held-out calibration set:

1. Train the model on the training data as usual.
2. Run it on n labeled calibration examples it has not seen and compute a nonconformity score for each, a number that grows as the true answer looks less typical to the model. For a classifier, a common score is 1 minus the probability assigned to the true class.
3. Take the ceil((n + 1) x (1 - alpha)) / n quantile of those scores, called qhat. The (n + 1) term is a finite-sample correction that makes the guarantee exact.
4. For a new input, include every candidate label whose score is at most qhat.

The resulting set contains the true label with probability at least 1 - alpha. Because the new example's score is exchangeable with the n calibration scores, it is equally likely to land at any rank among them, so it falls at or below qhat at least a 1 - alpha share of the time.

What the Guarantee Covers#

The guarantee is marginal: it holds on average over all inputs. A model can over-cover easy classes and under-cover a rare one, so class-conditional (Mondrian) conformal prediction calibrates a separate qhat per class. Exchangeability is the only assumption, and it fails under data drift, when production images differ from calibration images, so the calibration set has to come from the deployment data and be refreshed when that data changes. Dataset bias in the calibration set carries straight into the sets.

Set size is the useful signal. Every valid procedure reaches the target coverage, so methods compete on how small their sets are, and a more accurate model gives smaller sets. Adaptive Prediction Sets (APS) and Regularized Adaptive Prediction Sets (RAPS), the latter tested on ImageNet classifiers, use cumulative probabilities so set size tracks difficulty, and conformalized quantile regression produces intervals that widen where data is noisy.

Conformal prediction differs from three related ideas. A confidence score is a single number with no guarantee attached. Calibration methods such as temperature scaling make scores better match observed accuracy, which shrinks conformal sets but promises no coverage on its own. A statistical confidence interval describes uncertainty about a population parameter such as a mean, while a conformal set describes where a single new outcome will fall.

Prediction Sets in Practice#

The steps above in scikit-learn, with a deliberately small training set so the classifier makes mistakes:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Split handwritten digits into training, calibration, and test sets
X, y = load_digits(return_X_y=True)
X_train, X_rest, y_train, y_rest = train_test_split(X, y, train_size=100, random_state=0)
X_cal, X_test, y_cal, y_test = train_test_split(X_rest, y_rest, train_size=500, random_state=0)
model = LogisticRegression(max_iter=2000).fit(X_train, y_train)

# Nonconformity score: 1 minus the probability given to the true class
scores = 1 - model.predict_proba(X_cal)[np.arange(len(y_cal)), y_cal]

# Finite-sample corrected quantile for 95% coverage
alpha, n = 0.05, len(y_cal)
qhat = np.quantile(scores, np.ceil((n + 1) * (1 - alpha)) / n, method="higher")

# Prediction set: every class whose score is at most qhat
sets = 1 - model.predict_proba(X_test) <= qhat
coverage = sets[np.arange(len(y_test)), y_test].mean()
print(f"Top-1 accuracy: {model.score(X_test, y_test):.3f}")
print(f"Set coverage: {coverage:.3f}, average set size: {sets.sum(axis=1).mean():.2f}")

The top prediction alone is right 91.6% of the time, while the conformal sets contain the true digit for 96.5% of the 1,197 test images, above the 95% target, with an average of 1.27 labels. Libraries such as MAPIE package these steps for classification, regression, and time series.

Applications#

• Medical image analysis: A set of plausible findings with a known error rate tells clinicians in healthcare when a scan is ambiguous, and a set with one label marks the cases a model handles reliably.
• Large language models (LLMs): Conformal methods calibrate how many sampled answers to return or which claims to keep so that the output contains a correct answer at a chosen rate, complementing semantic entropy as a hallucination safeguard.
• Object detection: Conformal margins around predicted bounding boxes give a region that contains the true box at a chosen rate, useful for autonomous vehicles planning around pedestrians.

Image classifiers supply the same input as the scikit-learn example. Ultralytics YOLO26 classification models return the full softmax vector for every image in results[0].probs.data, so scoring a held-out set of labeled images gives qhat, and every class whose probability is at least 1 - qhat joins the prediction set.