Brier Score

The Brier score measures the accuracy of probabilistic predictions as the mean squared difference between each predicted probability and the actual outcome, coded as 1 if the event happened and 0 if it did not. For binary events it runs from 0, a perfect score, to 1, and a model that always predicts 0.5 scores exactly 0.25. Meteorologist Glenn W. Brier introduced it in 1950 to verify probabilistic weather forecasts, and it is now used to evaluate clinical risk models, forecasting tournaments, and machine learning (ML) classifiers. It scores the probabilities themselves, before any threshold turns them into decisions, so it separates a hesitant correct prediction of 0.51 from a confident one of 0.99, which accuracy, precision, and recall count the same.

How the Brier Score Is Calculated#

For N predictions with probability p and outcome o, the Brier score is the average of (p − o)². A model that predicts an 80% chance of a defect is penalized 0.04 if the defect is present, (0.8 − 1)², and 0.64 if it is not, (0.8 − 0)², so confident mistakes cost far more than hesitant ones. The Brier score is a strictly proper scoring rule: a model gets its best expected score only by reporting the probabilities it actually believes, so it cannot gain by shading predictions toward 0 or 1.

For more than two classes, the squared differences between the predicted probability vector and the one-hot vector of the true class are summed over classes and averaged over examples. That multi-class form, Brier's original definition, ranges from 0 to 2. The scikit-learn brier_score_loss function accepts both cases and by default halves the binary score to the usual 0-to-1 range while keeping the 0-to-2 range for multi-class input, so scores from different tools are comparable only with the same convention.

Interpreting a Brier Score#

A Brier score has no universal "good" value, because it depends on how often the event occurs. A model that always predicts the base rate p scores p(1 − p): 0.09 for an event that happens 10% of the time and 0.25 for one that happens half the time. The Brier skill score, BSS = 1 − BS / BS_ref, compares a model with such a reference forecast: 0 means no better than the reference, 1 is perfect, and negative values are worse than the reference.

from sklearn.metrics import brier_score_loss

# Outcomes (1 = defect) and the model's predicted defect probabilities
y_true = [0, 1, 1, 0, 1, 0, 0, 0, 1, 0]
y_proba = [0.10, 0.75, 0.60, 0.30, 0.90, 0.40, 0.05, 0.20, 0.35, 0.15]
model_score = brier_score_loss(y_true, y_proba)

# Reference forecast: always predict the observed defect rate
event_rate = sum(y_true) / len(y_true)
baseline_score = brier_score_loss(y_true, [event_rate] * len(y_true))

# Brier skill score: the share of the reference error the model removes
skill = 1 - model_score / baseline_score
print(f"Model {model_score:.3f}, baseline {baseline_score:.3f}, skill {skill:.2f}")

The script prints a model score of 0.098 against 0.240 for always predicting the 40% defect rate, a skill score of 0.59, meaning the model removes 59% of the reference forecast's squared error. Its largest single penalty, 0.42, comes from the defective part it rated at only 0.35.

Murphy's 1973 decomposition splits the score into three parts: BS = reliability − resolution + uncertainty. Reliability measures calibration, how far predicted probabilities sit from the observed frequencies of the cases that received them, and should be low. Resolution measures how much the predictions separate cases with different outcomes, and should be high. Uncertainty equals the base-rate term p(1 − p) and depends only on the data. A model that always predicts the base rate is perfectly calibrated with zero resolution, so its score equals the uncertainty term.

Brier Score vs. Related Metrics#

• Log loss: Also a strictly proper scoring rule, computed from the logarithm of the probability given to the true outcome. It punishes confident mistakes far harder: a 0.99 prediction for an event that does not happen costs 0.98 in Brier score and 4.6 in log loss, and a prediction of exactly 0 for an event that happens makes log loss infinite.
• Expected calibration error (ECE): Measures calibration alone by binning predictions by confidence and averaging the gap between confidence and accuracy. It ignores resolution, so a model that always predicts the base rate can score a perfect ECE.
• Area under the curve (AUC): Measures how well scores rank positives above negatives and is unchanged by any calibration that preserves that order, such as temperature scaling of a binary classifier.
• Confidence: The score a model attaches to a prediction. The Brier score tests whether those scores behave as probabilities, and overconfident scores raise it through the reliability term.

Brier Score in Computer Vision#

Image classifiers that output probabilities are scored with the Brier score on labeled held-out images, and recomputing it on newly labeled production images shows whether calibration holds as inputs change through data drift. YOLO26 classification models return a full softmax distribution over their classes in result.probs.data, so the multi-class Brier score of a fine-tuned YOLO26 classifier is the mean, over held-out images, of the summed squared differences between that vector and the one-hot true label.