Data Leakage

Data leakage in machine learning (ML) occurs when a model's training data or evaluation includes information it will not have when it makes real predictions, such as examples from the test set, values from the future, or features that encode the answer. The model appears accurate during development and model testing, then underperforms in production, because the shortcut it relied on is gone. Leakage is a leading cause of irreproducible results: a survey of machine-learning-based science counted 294 affected papers across 17 fields, and in civil war prediction, complex models lost their reported advantage over decades-old logistic regression once the leakage was corrected.

How Data Leakage Happens#

Leakage enters through the way data is split, processed, or chosen as input, and the same mistakes recur in tabular models, time series analysis, and vision tasks such as action recognition:

• Train-test contamination: Steps that learn from data, such as normalization statistics, imputation, or feature selection, are fitted on the whole dataset before splitting, so the training data carries information about the evaluation split. Duplicate images and augmented copies made before splitting have the same effect.
• Group leakage: Related samples are split apart, such as consecutive frames of one video, several scans of one patient, or photos of one scene. A model can then score well by recognizing the scene or person instead of learning the task.
• Temporal leakage: Randomly splitting time-ordered data, or computing features with values from after the prediction time, lets a model look into the future.
• Target leakage: An input is a consequence or proxy of the label, such as a "received treatment" field in a model that predicts disease.
• Evaluation reuse: Tuning against the test set, or benchmark questions appearing in the web-scale pretraining data of a large language model, turns the evaluation into training signal.

Images can carry target leakage too. In the ChestX-ray14 dataset, 80% of test-set pneumothorax X-rays also showed a chest drain, the tube used to treat the condition. A classifier trained on the dataset reached an ROC AUC of 0.94 on images with a drain but only 0.77 on those without one, the untreated cases a detector exists to catch.

Demonstrating Leakage in Code#

Fitting a data-dependent step before cross-validation is the textbook mistake, and its effect is easy to measure. The following example generates pure noise, so no model can beat 50% accuracy on new data, then selects the 20 features that best separate the labels and cross-validates a logistic regression twice: once with selection fitted on all samples, and once with selection inside a scikit-learn pipeline that refits it on each training fold.

import numpy as np
from sklearn.feature_selection import SelectKBest
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

# Pure noise: 100 samples, 5,000 random features, random labels (true accuracy is 50%)
rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))
y = rng.integers(0, 2, size=100)

# Leaky: pick the 20 features that best separate the labels using every sample, then cross-validate
X_selected = SelectKBest(k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(), X_selected, y, cv=5).mean()

# Correct: feature selection is refit inside each training fold
pipeline = make_pipeline(SelectKBest(k=20), LogisticRegression())
clean = cross_val_score(pipeline, X, y, cv=5).mean()
print(f"Leaky CV accuracy: {leaky:.2f}, correct CV accuracy: {clean:.2f}")

The leaky version reports 85% accuracy on random labels, while the pipeline reports 48%, which is chance. With 5,000 features and 100 samples, some features correlate with the labels by luck, and choosing them with the validation folds included builds that luck into every fold's score.

Preventing Data Leakage#

Prevention starts with the split, which comes before any step that learns from data, so the validation and test splits stay unseen:

• Split before feature engineering: Fit scalers, imputers, encoders, and feature selectors on the training split only, and apply them unchanged to the other splits.
• Split by group or time: Keep every sample from one patient, video, or scene in a single split, and train on the past to test on the future.
• Deduplicate with hashes and embeddings: Compare file hashes for exact copies and embedding similarity for near-duplicates across splits before training.
• Audit inputs: Ask whether each feature exists at the moment of prediction, and treat a single dominant feature or a suspiciously high score as a bug to investigate.

Leakage also distorts checkpoint selection. Ultralytics YOLO learns from the train images listed in a dataset YAML and saves best.pt from the epoch with the highest validation fitness, which is mAP50-95 for detection, so near-duplicate frames shared between train and val inflate both the score that picks the checkpoint and the score that is reported.

Data Leakage vs. Related Terms#

• Overfitting: The model memorizes noise in legitimate training data and scores poorly on a clean validation set. Leakage makes the validation score itself wrong, so an overfit model can look good.
• Shortcut learning: The model relies on a spurious cue, such as a watermark or background, that correlates with the label. Leakage is the case where the cue reaches training or evaluation from information unavailable in deployment.
• Data security: In cybersecurity, data leakage means sensitive data exposed to unauthorized parties, which access controls, encryption, and data loss prevention tools address. The two meanings share a name only.