Cross-Validation

Cross-validation estimates how well a model will perform on unseen data by training and evaluating it several times on different splits of the same dataset and combining the scores. In k-fold cross-validation, the data is divided into k equal parts called folds, and each fold serves once as validation data while the model trains on the other k − 1 folds. The mean of the k scores estimates performance, and their spread shows how much that estimate depends on which examples happened to land in the validation set. Cross-validation is the standard evaluation method when data is too scarce for a single held-out split to give a reliable score, and it is widely used to compare algorithms and tune hyperparameters without touching the test data.

How K-Fold Cross-Validation Works#

The dataset is shuffled once and split into k folds, usually 5 or 10. In each of k rounds, a fresh model is trained on k − 1 folds and scored on the remaining fold, so every example is used for validation exactly once and for training k − 1 times. The k scores are averaged and reported with their standard deviation. Once cross-validation has chosen a model or a set of hyperparameters, the final model is usually retrained on all of the training data with those settings. Leave-one-out cross-validation is the extreme case where k equals the number of examples, which uses the most data for training in each round at the cost of training as many models as there are examples.

Reading Cross-Validation Scores#

The spread across folds is as informative as the mean. The scikit-learn script below runs 5-fold cross-validation on the bundled breast cancer dataset:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# 569 labeled tumor samples with 30 features each
X, y = load_breast_cancer(return_X_y=True)

# The scaler sits inside the pipeline, so each round fits it on that round's training folds only
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

# 5 stratified folds: each sample is validated exactly once
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=folds)
print("Fold accuracies:", scores.round(3))
print(f"Mean {scores.mean():.3f} +/- {scores.std():.3f}")

The five fold accuracies range from 0.956 to 1.000, with a mean of 0.979 and a standard deviation of 0.014. A single 80/20 split would have reported any one of those five numbers, a range of 4.4 percentage points. Placing the scaler inside the pipeline keeps the validation fold out of the data preprocessing statistics; scaling the full dataset first would cause data leakage from each validation fold into training.

Choosing a Cross-Validation Variant#

Plain k-fold assumes examples are independent and interchangeable. When they are not, a random split leaks information or produces folds that misrepresent the task:

• Stratified k-fold: Keeps each fold's class proportions equal to the full dataset's, which matters when a rare class could otherwise be missing from a fold.
• Group k-fold: Keeps all examples from one group, such as one patient, video, or field site, in the same fold, so the model is never validated on near-duplicates of its training images. Stratified group k-fold combines both constraints.
• Time series split: Trains only on the past and validates on the period that follows, expanding the training window each round, because shuffling time-ordered data lets the model see the future.
• Nested cross-validation: Runs an inner cross-validation loop for hyperparameter tuning inside each outer fold, so the outer score is not inflated by tuning on the same folds it reports.

Cross-Validation for Deep Learning#

Training a deep network k times multiplies the compute cost by k, so large datasets such as ImageNet and COCO use a single fixed validation split, which is already big enough for a stable score. Cross-validation pays off for small vision datasets of a few hundred to a few thousand images, where one validation split may hold only a handful of examples of each class. The Ultralytics K-Fold Cross Validation guide splits a detection dataset into five folds with scikit-learn's KFold, writes training and validation image lists and a dataset YAML for each fold, and trains one YOLO26 model per fold. Because detection images usually contain several classes, they cannot be stratified by a single label, so the guide also records the ratio of validation to training labels for each class in each fold.

Cross-Validation vs. Related Terms#

• Hold-out validation: A single split, such as 80/20, scored once. It costs one training run and suits large datasets, and its score depends on which examples fall in the validation split.
• Bootstrapping: Draws training sets by sampling with replacement and scores each model on the examples left out of its draw. K-fold partitions the data without replacement, so each example sits in exactly one fold.
• Test set evaluation: Cross-validation chooses among models and settings. A separate test set, kept outside all folds, gives the final estimate for the chosen model.