Fine-tuning

Fine-tuning is the process of continuing to train a model that was already trained on a large dataset, updating some or all of its weights on a smaller, task-specific dataset. The pretrained model weights already encode general knowledge, such as edges, textures, and shapes in vision models or grammar and facts in large language models (LLMs), so fine-tuning reaches high accuracy with far less labeled data and compute than training from random weights. It is the most common way to carry out transfer learning, and it is how most custom object detection models and most chat assistants are built from pretrained foundation models.

How Fine-Tuning Works#

A fine-tuning run loads pretrained weights, replaces or resizes any layer tied to the original task, such as a classifier or detection head, and trains on the new data with the usual loss and optimizer. Three choices decide how much of the original model survives:

• Learning rate: Fine-tuning uses a lower learning rate than pretraining, usually after a short warmup, so updates refine pretrained features instead of overwriting them. ULMFiT went further with discriminative learning rates, dividing the rate by 2.6 for each layer below the top.
• Which layers train: Freezing the early layers of the backbone keeps general features fixed, cuts memory use, and lowers the risk of overfitting on small datasets. Gradual unfreezing trains the new head first, then releases deeper layers.
• How long to train: The longer a model trains only on new data, the more of its original skills it loses to catastrophic forgetting.

The right setup depends on the available memory, the amount of labeled data, and whether the original skills still matter:

Full vs. Parameter-Efficient Fine-Tuning#

Full fine-tuning updates every weight, which for a 7-billion-parameter LLM means holding gradients and optimizer state for 7 billion values and saving a complete model copy for every task. Parameter-efficient fine-tuning (PEFT) freezes the pretrained weights and trains a small set of added or selected parameters instead, and Hugging Face's PEFT library implements the common methods. LoRA adds trainable low-rank matrices beside frozen weight matrices and, compared with fully fine-tuning GPT-3 175B with Adam, cut trainable parameters by 10,000 times and GPU memory by 3 times. QLoRA trains LoRA adapters on top of a 4-bit quantized base model, which made fine-tuning a 65B-parameter model possible on a single 48 GB GPU.

LLMs pass through several fine-tuning stages after pretraining. Instruction tuning trains on prompt and response pairs, and preference-based methods such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) then align the responses with human preferences.

Fine-Tuning vs. Related Terms#

• Transfer learning: The broader idea of reusing a model trained on one task for another. Fine-tuning updates the pretrained weights, while feature extraction keeps them frozen and trains only a new head.
• Training from scratch: Starts from random weights and needs far more labeled data and compute. It suits new architectures and domains with little in common with any available pretraining data.
• Linear probing: Trains only a linear classifier on frozen features, usually to measure how much a pretrained representation already captures.
• Prompt engineering and retrieval-augmented generation (RAG): Change an LLM's behavior through its input, leaving the weights untouched.

Fine-Tuning an Object Detector#

Calling train() on a pretrained YOLO26 checkpoint with a new dataset fine-tunes it: every compatible weight transfers, and the default cls_remap=True copies classifier rows whose class names match, so on the African Wildlife dataset the elephant and zebra outputs start from their COCO-pretrained rows, while buffalo and rhino, which COCO lacks, start fresh.

from ultralytics import YOLO

# Load COCO-pretrained YOLO26n weights; YOLO("yolo26n.yaml") would train from scratch instead
model = YOLO("yolo26n.pt")

# Fine-tune on 1,052 training images of buffalo, elephants, rhinos, and zebras
results = model.train(data="african-wildlife.yaml", epochs=30, imgsz=640)

For this 30-epoch run, the default optimizer="auto" selects AdamW with lr0=0.00125, a rate it computes from the dataset's 4 classes, because the run stays under 10,000 iterations.