Validation Data

Validation data is a portion of labeled data held out from training and scored repeatedly during development to compare models, tune hyperparameters, pick checkpoints, and detect overfitting. The model never updates its weights on it, so its validation score shows how well what it learned from the training data carries over to images it has not seen. Validation set and development (dev) set are synonyms. Every decision based on validation scores fits the model a little more closely to that particular split, which is why a separate test set is kept back for the final, unbiased estimate.

Training, Validation, and Test Splits#

A labeled dataset is usually divided into three parts:

• Training set: The examples the optimizer fits the weights to, typically 70 to 80 percent of the data.
• Validation set: Scored after training runs or epochs to guide choices such as the learning rate, model size, augmentation strength, and when to stop.
• Test set: Scored once, after all choices are fixed, to report expected real-world performance.

Splits of 70/15/15 or 80/10/10 are common for datasets of thousands of images. Very large datasets need a smaller share for reliable estimates: ImageNet-1k holds out 50,000 validation images against 1,281,167 training images. Because its test labels are not public, ImageNet accuracy is conventionally reported on that validation set, which then serves as a test set.

A useful validation set matches the data the model will meet in deployment and shares nothing with the training set. Random image-level splits leak information when several images come from the same video, patient, or location, because the model can recognize the scene instead of the object. Splitting by group, such as by patient or recording session, prevents this data leakage.

Hyperparameter Tuning and Model Selection#

Weights are learned by gradient descent, while hyperparameters are chosen by training several candidates and keeping the one with the best validation score. Training scores cannot make this choice, because larger and longer-trained models nearly always fit their training data better. The scikit-learn script below tunes the depth of a decision tree on the bundled digits dataset:

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

# Split 1,797 digit images 60/20/20 into training, validation, and test sets
X, y = load_digits(return_X_y=True)
X_train, X_rest, y_train, y_rest = train_test_split(X, y, test_size=0.4, random_state=0, stratify=y)
X_val, X_test, y_val, y_test = train_test_split(X_rest, y_rest, test_size=0.5, random_state=0, stratify=y_rest)

# Fit one model per candidate tree depth and score each on the validation set
val_scores = {}
for depth in (2, 4, 8, 16):
    model = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    val_scores[depth] = model.score(X_val, y_val)
    print(f"depth {depth}: train {model.score(X_train, y_train):.3f}, validation {val_scores[depth]:.3f}")

# Refit the winner and score the test set exactly once
best = max(val_scores, key=val_scores.get)
final = DecisionTreeClassifier(max_depth=best, random_state=0).fit(X_train, y_train)
print(f"Chosen depth {best}: test accuracy {final.score(X_test, y_test):.3f}")

Training accuracy keeps rising with depth and reaches 1.000 at depth 16, while validation accuracy peaks at 0.836 at depth 8 and falls to 0.825 at depth 16 as the tree starts memorizing. Validation picks depth 8, which scores 0.853 on the untouched test set.

Early Stopping and Checkpoint Selection#

Deep learning training loops usually score the validation set after every epoch. Training loss keeps falling as the network fits its training data ever more closely, but validation loss or accuracy eventually levels off and then worsens. Early stopping ends training once the validation score has stopped improving for a set number of epochs, called the patience, and keeps the weights from the best epoch.

YOLO26 training in Ultralytics runs this loop by default: it scores the val split from the dataset YAML after every epoch, saves the epoch with the highest validation fitness (mAP_50-95 for detection) as best.pt, and stops after patience epochs without improvement, 100 by default.

from ultralytics import YOLO

# Load a pretrained YOLO26 nano detector
model = YOLO("yolo26n.pt")

# Validate on the val split after each epoch and stop after 10 epochs without improvement
results = model.train(data="coco8.yaml", epochs=50, patience=10)

Cross-Validation for Small Datasets#

When data is scarce, a single held-out split wastes examples and gives a noisy score. K-fold cross-validation divides the data into K folds, trains K models that each hold out a different fold for validation, and averages the K scores, so every example is used for both training and validation. Five and ten folds are common, and stratified and grouped variants keep class proportions balanced or related images together within each fold. A test set still sits outside all folds.

Validation Data vs. Related Terms#

• Test data: Scored once at the end. Reusing it to make choices turns it into a second validation set, and its score stops being an unbiased estimate.
• Training data: The only split the optimizer learns weights from.
• Cross-validation: A procedure that rotates which examples serve as validation data, producing K validation scores from one dataset.
• Validation in clinical and regulatory writing: In medicine and regulated industries, validating a model often means evaluating the finished model on independent external data, which corresponds to the test set in machine learning terminology.