Transfer Learning

Transfer learning is a machine learning technique that reuses a model trained on one task as the starting point for a related task. Training a neural network from random weights takes large labeled datasets and substantial compute, while a model pretrained on millions of images has already learned visual features that most new tasks need. Starting from those learned model weights, a new model reaches high accuracy from far fewer labeled images and in a fraction of the training time, which is why transfer learning is the default way to build computer vision models for custom tasks.

How Transfer Learning Works#

Transfer learning works because deep networks learn features in a hierarchy. The first layers of a backbone respond to edges, colors, and textures that appear in almost any image, while deeper layers combine them into object parts and task-specific concepts. A study by Yosinski et al. measured this progression in convolutional networks, finding that first-layer features are general, that transferability falls in higher layers as features specialize to the original task, and that transferred features improve generalization even after fine-tuning on the new dataset.

The process has two phases:

1. Pretraining: A model learns general representations on a large benchmark dataset such as ImageNet, COCO, or Objects365, producing a set of pretrained weights.
2. Adaptation: The output layers, such as a classifier or detection head, are replaced or resized for the new classes and trained on the target dataset. In feature extraction, the pretrained layers stay frozen and only the new layers learn. In fine-tuning, some or all pretrained layers also update, usually at a lower learning rate.

How much of the network to retrain depends on two questions: how much labeled data the new task has, and how closely it resembles the pretraining data. Stanford's CS231n notes lay out the four resulting cases:

Transfer Learning for Object Detection#

Detection models are almost always trained by transfer: the official YOLO26 checkpoints were pretrained on Objects365 for 150 epochs before fine-tuning on COCO, as the YOLO26 training recipe documents. Calling train() on a loaded checkpoint with a new dataset transfers every compatible weight, resets the class-specific output rows when the number of classes changes while keeping rows whose class names match, and accepts a freeze argument that holds early layers fixed, which helps most on small datasets.

from ultralytics import YOLO

# Load YOLO26n weights pretrained on COCO's 80 classes
model = YOLO("yolo26n.pt")

# Adapt to a new single-class dataset; freeze=10 keeps the first 10 layers fixed
results = model.train(data="medical-pills.yaml", epochs=20, freeze=10)

The pretrained backbone already detects shapes, edges, and textures, so training only has to learn what a pill looks like from the 92 labeled training images in the Medical Pills dataset.

Transfer Learning vs. Related Terms#

• Fine-tuning: One way to carry out transfer learning, in which pretrained weights keep updating on the new data. Feature extraction with a frozen backbone is the other main way.
• Domain adaptation: A form of transfer learning where the task stays the same but the data distribution shifts, such as from daytime to nighttime images, often with few or no target labels.
• Zero-shot learning and few-shot learning: Handle new classes with no labeled examples or only a handful, relying on a pretrained model's general knowledge instead of a full adaptation phase.
• Parameter-efficient fine-tuning (PEFT): Adapts very large pretrained models by training a small set of added or selected parameters, such as LoRA matrices, while the original weights stay frozen.
• Catastrophic forgetting: The loss of the original task's skills when a model is fine-tuned on new data. Mixing some original-class images into the new training set, or freezing more layers, limits it.