Brier Score

The Brier Score measures the accuracy of probabilistic predictions by averaging the squared difference between each predicted probability and the outcome that actually occurred. A score of 0 represents perfect predictions; larger values indicate that predictions were less accurate, more poorly calibrated, or both. Unlike metrics based only on final class labels, the Brier Score evaluates whether a machine learning model assigns sensible probabilities.

How the Brier Score Works#

For binary classification, encode the outcome as 1 when an event occurs and 0 when it does not. For every example, calculate (predicted probability - outcome) squared, then average those values.

Suppose a model predicts an 80% probability that an image contains a defect:

• If the defect is present, the squared error is (0.8 - 1) squared, or 0.04.
• If no defect is present, the error is (0.8 - 0) squared, or 0.64.

The second prediction receives a much larger penalty because the model was confidently wrong. This makes the Brier Score a strictly proper scoring rule: on average, a forecaster obtains the best score by reporting its honest probability estimate. The scikit-learn Brier score loss documentation provides a standard implementation.

For binary problems, the conventional range is 0 to 1. Multiclass versions sum the squared errors across every class, so their range may be 0 to 2 unless divided by two. Because libraries use different scaling conventions, such as the rescaled approach in yardstick’s multiclass Brier Score, comparisons should use the same implementation and settings.

Interpreting a Good Brier Score#

Lower is better, but there is no universal threshold for a “good” Brier Score. Its meaning depends on event prevalence, dataset difficulty, and the quality of a reasonable baseline.

For example, if an event occurs in 10% of cases, a model that always predicts 0.10 receives an expected score of 0.09. A useful model should generally improve on that prevalence-based baseline. Evaluation should use representative validation data, with separate reporting for important classes, operating environments, or demographic groups.

The score reflects two related properties:

• Calibration: Predictions of 0.80 should be correct approximately 80% of the time. Probability calibration methods and reliability diagrams can reveal systematic overconfidence or underconfidence.
• Resolution: Predictions should meaningfully separate likely events from unlikely ones. A model that always predicts the base rate may be calibrated overall but provides little case-specific information.

A single aggregate score can hide local problems. Pair it with a calibration curve, subgroup analysis, and broader uncertainty quantification.

Brier Score vs. Related Metrics#

• Accuracy, precision, and recall: These evaluate discrete decisions after applying a threshold. The Brier Score evaluates the probabilities before thresholding, distinguishing a cautious correct prediction of 0.51 from a confident one of 0.99.

• ROC AUC: AUC measures ranking—whether positives tend to receive higher scores than negatives. A model can rank cases correctly yet produce poorly calibrated probabilities.

• Log loss: Both are proper scoring rules, but log loss penalizes extremely confident mistakes more sharply. The Brier Score’s squared-error interpretation is often easier to explain.

• Confidence: A model output labeled “confidence” is not automatically a calibrated event probability. Its meaning must be validated before applying the Brier Score.

Real-World Applications#

In medical image triage, an image classification model might estimate the probability that a scan contains an abnormality. The Brier Score can evaluate whether those probabilities match observed diagnoses, which matters when referral rules treat a 90% estimate differently from a 55% estimate.

In manufacturing visual inspection, a model may estimate the probability that a product is defective. Well-calibrated predictions support risk-based routing: high-probability cases can be rejected, uncertain cases sent for human inspection, and low-probability cases accepted. Poor calibration can cause escaped defects or unnecessary waste. The NIST AI Risk Management Framework Core emphasizes measuring uncertainty and monitoring performance in deployed systems.

Practical Evaluation Workflow#

The following example calculates a model’s score and compares it with a constant prevalence baseline:

from sklearn.metrics import brier_score_loss

# Binary outcomes and predicted event probabilities
y_true = [0, 1, 1, 0, 1, 0]
y_probability = [0.10, 0.75, 0.60, 0.30, 0.90, 0.40]

model_score = brier_score_loss(y_true, y_probability)

# Compare against always predicting the observed event rate
event_rate = sum(y_true) / len(y_true)
baseline_probability = [event_rate] * len(y_true)
baseline_score = brier_score_loss(y_true, baseline_probability)

print(f"Model: {model_score:.3f}")
print(f"Baseline: {baseline_score:.3f}")

For computer vision, Ultralytics YOLO26 classification predictions expose class probabilities through Predict mode. Collect these probabilities on labeled held-out images before calculating the score. Combine the result with Ultralytics validation mode and the YOLO performance metrics guide rather than treating it as a replacement for task-specific metrics.

After deployment, recalculate the score when new ground-truth labels become available. Ultralytics Platform deployment monitoring can support the surrounding operational workflow, while recurring labeled evaluations help detect calibration changes caused by data drift.