Data Drift

Data drift is a change in the statistical distribution of the input data a deployed model receives, relative to the data it was trained on. Models assume that production inputs come from the same distribution as their training data, and when that assumption breaks, accuracy can fall with no error message and no change to the model's code or weights. Common causes in vision systems include new cameras or lenses, changes in lighting and seasons, a moved camera angle, new product packaging, and upstream pipeline changes such as a different image compression setting. Detecting and responding to drift is a core task of model monitoring and machine learning operations (MLOps).

Types of Distribution Shift#

A classifier models the probability of a label given an input, and the shift can occur in the inputs, the labels, or the relationship between them:

• Covariate shift: The input distribution changes while the correct label for any given input stays the same. A detector trained on daytime images that starts receiving dusk footage faces covariate shift, because a car is still a car. Data drift in the narrow sense means this case.
• Label shift: Also called prior probability shift, the class frequencies change. If a production line's defect rate rises from 1% to 5%, the share of a detector's alarms that are real rises with it, so its precision changes even though the model and the look of each defect are unchanged.
• Concept drift: The relationship between inputs and labels changes, so the same input now deserves a different label. Fraud patterns evolve as fraudsters adapt, and a quality standard that tightens turns parts once accepted into rejects.

Drift can be sudden, such as a camera replacement, gradual, such as lens dirt building up, or recurring, such as seasonal foliage and snow.

Detecting Data Drift#

Drift detection compares a reference sample, usually the training or validation data, with a recent window of production data. Without labels, teams test whether the two input distributions differ, using the two-sample Kolmogorov-Smirnov (KS) test for continuous features, a chi-squared test for categorical ones, or the population stability index (PSI) common in credit scoring. Raw pixels are too high-dimensional to compare directly, so image monitoring tests summary statistics such as brightness and sharpness, or the embeddings of a pretrained network; a study of shift-detection methods found two-sample tests on representations from pretrained classifiers worked best. The model's own outputs are a second signal: shifts in predicted class frequencies, boxes per image, or confidence scores often appear before anyone measures accuracy.

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(0)

# Mean brightness (0-255) of 500 training images and two batches of 200 production images
reference = rng.normal(130, 25, 500)
same_light = rng.normal(130, 25, 200)
dusk = rng.normal(110, 25, 200)

# Two-sample Kolmogorov-Smirnov test: a small p-value means the distributions differ
for name, batch in (("same light", same_light), ("dusk", dusk)):
    result = ks_2samp(reference, batch)
    drifted = result.pvalue < 0.01
    print(f"{name}: KS statistic {result.statistic:.3f}, p-value {result.pvalue:.2g}, drift {drifted}")

The batch drawn from the training distribution scores a KS statistic of 0.046 with a p-value of 0.91, and the batch whose mean brightness fell by 20 levels scores 0.346 with a p-value of 1.1e-15 and is flagged. The KS statistic is the largest gap between the two cumulative distributions, and with very large samples even harmless shifts reach tiny p-values, so drift alerts are often set on the KS statistic itself, which measures the size of the shift, instead of the p-value alone.

Responding to Drift#

Only fresh labels show whether a detected shift has lowered accuracy. The standard response is to sample recent production data, label it, measure mean average precision (mAP) or other task metrics on it, and retrain or fine-tune on a mix of new and original data when performance has dropped, keeping old examples in the mix to avoid catastrophic forgetting. Data augmentation such as brightness, blur, and color changes prepares a model for variation that can be anticipated, and unsupervised domain adaptation adjusts a model to a new domain with unlabeled target data. On Ultralytics Platform, the Monitoring tab of a dedicated endpoint charts mean prediction confidence over time, class counts, and box sizes and locations for the images it serves, and recent examples can be saved to a dataset for review and retraining.

Data Drift vs. Related Terms#

• Model drift: The decline in a model's performance over time, the effect that data drift and concept drift cause. Also called model decay.
• Training-serving skew: A mismatch between training and production data caused by different processing in the two pipelines, such as another resize method, present from the first day of deployment rather than developing over time.
• Anomaly detection: Flags individual inputs that differ from the norm, such as a single corrupted frame. Drift detection compares whole distributions and flags a week of uniformly darker images.
• Out-of-distribution (OOD) detection: Scores each input for how unlike the training data it is, usually so the model can abstain or defer to a person on that input.