Parameter-Efficient Fine-Tuning (PEFT)

Parameter-efficient fine-tuning (PEFT) adapts a pretrained model to a new task by training only a small fraction of its parameters, either a few new ones added to the network or a chosen subset of the existing ones, while the rest of the model weights stay frozen. Full fine-tuning updates every weight, which for a 7-billion-parameter large language model (LLM) means holding gradients and optimizer state for all 7 billion values and saving a complete copy of the model for every task. PEFT methods often train less than 1% of the parameters, so a foundation model can be adapted on a single GPU, and each new task adds a small file of trained parameters instead of another full checkpoint. Hugging Face's PEFT library implements the most common methods for transformer models.

How PEFT Works#

PEFT builds on transfer learning: a model pretrained on broad data already encodes general features, so adapting it to a related task changes far less than training it did. Methods fall into three families, depending on which parameters they train.

Additive methods insert new trainable modules or inputs into the frozen network. Adapters add small bottleneck layers inside each transformer block and came within 0.4% of full fine-tuning on the GLUE language benchmark while adding 3.6% parameters per task. Prompt tuning and prefix tuning learn a handful of continuous vectors that are prepended to the input or to each layer's activations.

Selective methods train a chosen subset of the existing weights. Freezing the backbone and training only the later layers is the simplest form, and BitFit trains only the bias terms.

Reparameterization methods learn a compact update to frozen weight matrices. LoRA represents each update as the product of two low-rank matrices and, compared with fully fine-tuning GPT-3 175B, cut trainable parameters by 10,000 times and GPU memory by 3 times. QLoRA trains LoRA adapters on top of a 4-bit quantized base model, and DoRA splits each weight into a magnitude and a direction and adapts the direction with LoRA.

Because only the small trained parameters differ between tasks, one copy of the base model can serve many tasks at once. A server keeps the frozen base in memory and applies the adapter that matches each request, and systems such as S-LoRA serve thousands of LoRA adapters this way:

Benefits and Trade-Offs#

Gradients and optimizer state are needed only for the trained parameters, so memory use drops sharply, although the frozen weights and the activations still have to fit on the GPU. Keeping the pretrained weights fixed also limits overfitting on small datasets and catastrophic forgetting of the model's original skills. The cost is capacity: in a study titled LoRA Learns Less and Forgets Less, LoRA trailed full fine-tuning on programming and mathematics while preserving more of the base model's performance on tasks outside those domains.

PEFT in Computer Vision#

Vision transformers adapt with the same methods. Visual Prompt Tuning learns extra input tokens for a frozen ViT, training less than 1% of the parameters, and outperformed full fine-tuning in many of its tests, while LoRA adapters customize Stable Diffusion to a new style or subject. For convolutional object detection models, freezing is the standard selective approach. In Ultralytics YOLO26 training, the freeze argument stops gradient updates for the first N layers and also fixes their batch normalization statistics, and freeze=11 covers layers 0 to 10, the whole YOLO26 backbone, so only the neck and detection head train:

from ultralytics import YOLO

# Load COCO-pretrained YOLO26n weights
model = YOLO("yolo26n.pt")

# Freeze layers 0-10 (the backbone) and train only the neck and detection head
results = model.train(data="coco8.yaml", epochs=5, freeze=11)

The training log lists each frozen parameter on a Freezing layer line before the first epoch starts.

PEFT vs. Related Terms#

• Full fine-tuning: Updates every weight, which gives the most capacity to adapt but needs the most memory and a full model copy per task. PEFT is a form of fine-tuning that restricts which parameters change.
• Linear probing: Trains only a linear classifier on frozen features, the most restrictive selective method, and is mainly used to measure how much a pretrained representation already captures.
• Prompt engineering: Changes a model's behavior through handwritten text in its input and trains nothing. Prompt tuning learns continuous prompt vectors by gradient descent.
• Knowledge distillation: Trains a smaller student model to imitate a larger teacher, which compresses a model, while PEFT adapts an existing model to a new task.
• Model merging: Combines trained weights or adapters into one model. A trained LoRA update can be added into the frozen weight matrix it adapts, so the merged model runs with no extra inference cost.