Linear Probing

Linear probing is a technique that measures how much useful information a pretrained model's representations contain by training only a linear classifier, the probe, on top of its frozen features. The encoder's weights never change: images or text pass through it once, the resulting embeddings become fixed inputs, and a logistic regression or single dense layer learns to predict labels from them. High probe accuracy means the encoder already arranges examples so that a straight boundary separates the classes. In data structures, the same name refers to an unrelated method for resolving collisions in hash tables by checking the next slot.

How Linear Probing Works#

A probe experiment loads a pretrained encoder, freezes it, runs feature extraction on a labeled dataset, and trains a linear classifier on the stored features, with results reported on a held-out test set. The probe has one weight per feature per class plus a bias, so it can only draw linear decision boundaries and cannot build new features of its own; any accuracy it reaches reflects what the encoder learned. Logistic regression applies a softmax or sigmoid to its scores, but the decision boundaries remain linear in the features. Because only the probe trains, the features can be computed once and reused, which makes probing far cheaper than fine-tuning.

Ultralytics YOLO26 classification models expose frozen features through model.embed(), which by default pools the output of the layer before the classification head into one vector per image, 256 numbers for the nano model. The following example probes those features on CIFAR-10 cats and dogs, low-resolution images the ImageNet-trained model never saw, and fits the same probe on raw pixels for comparison:

from pathlib import Path

import numpy as np
from PIL import Image
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from torchvision.datasets.utils import download_and_extract_archive
from ultralytics import YOLO

# 500 cat and 500 dog images from the CIFAR-10 test split
download_and_extract_archive("https://github.com/ultralytics/assets/releases/download/v0.0.0/cifar10.zip", "data")
cats = sorted(Path("data/cifar10/test/cat").glob("*.png"))[:500]
dogs = sorted(Path("data/cifar10/test/dog").glob("*.png"))[:500]
paths, labels = cats + dogs, [0] * 500 + [1] * 500

# Frozen YOLO26 classifier features, and raw pixels as a baseline representation
encoder = YOLO("yolo26n-cls.pt")
features = np.stack([encoder.embed(path, verbose=False)[0].cpu().numpy() for path in paths])
pixels = np.stack([np.asarray(Image.open(path)).ravel() / 255 for path in paths])

# Train the same linear probe on each representation and score it on held-out images
for name, x in [("raw pixels", pixels), ("YOLO26 features", features)]:
    x_train, x_test, y_train, y_test = train_test_split(x, labels, test_size=0.3, random_state=0, stratify=labels)
    probe = LogisticRegression(max_iter=5000)
    probe.fit(x_train, y_train)
    print(f"{name}: {probe.score(x_test, y_test):.3f}")

The probe scores 0.573 on raw pixels, barely above the 0.5 of guessing, and 0.833 on YOLO26 features, a gap that measures the class information the frozen YOLO26 encoder makes linearly accessible.

What a Probe Reveals#

A linear probe tests whether information is both present and easy to read out. Guillaume Alain and Yoshua Bengio proposed linear classifier probes as a diagnostic in "Understanding Intermediate Layers Using Linear Classifier Probes" and found that the linear separability of features increased monotonically with depth in the networks they studied. Probing every layer this way shows where a network forms the concepts a task needs. In self-supervised learning, linear-probe accuracy on ImageNet became the standard way to compare encoders trained without labels, since the probe adds almost nothing a weak representation could lean on. In natural language processing, probes trained on a language model's hidden states test whether it encodes properties such as part of speech, syntax, or whether a statement is true.

Probe results depend on the layer chosen, input resolution and preprocessing, the amount of labeled data, the probe's regularization, and the split between training and evaluation images, so comparisons between encoders hold only when all of these stay fixed. When a linear probe fails, a small nonlinear probe sometimes succeeds, which shows the information is present but encoded nonlinearly.

Linear Probing vs. Related Methods#

• Fine-tuning: Updates some or all encoder weights along with the head, which usually gives higher accuracy on the target data. In "Fine-Tuning Can Distort Pretrained Features", fine-tuning averaged 2% higher in-distribution accuracy than linear probing across 10 datasets but 7% lower accuracy under distribution shift, and LP-FT, linear probing first and then fine-tuning from that head, beat full fine-tuning by 1% in-distribution and 10% under shift.
• k-nearest neighbors evaluation: Classifies each test embedding by the labels of the closest training embeddings with no training at all, another common protocol for self-supervised encoders.
• Feature extraction and transfer learning: Feature extraction produces the embeddings a probe consumes, and training a linear head on a frozen backbone is the cheapest form of transfer learning, which PyTorch's transfer learning tutorial calls using the network as a fixed feature extractor.

Limits of Linear Probing#

Linear probing measures how linearly accessible information is, which differs from how useful a representation is after adaptation. The authors of masked autoencoders (MAE) found that linear probing and fine-tuning results were largely uncorrelated: their ViT-L encoder reached 73.5% ImageNet accuracy under linear probing, fine-tuning only its last transformer block raised that to 81.0%, and MoCo v3 probed higher than MAE yet trailed it in every partial fine-tuning setting.