LoRA (Low-Rank Adaptation)

LoRA, short for Low-Rank Adaptation, is a parameter-efficient fine-tuning (PEFT) method that adapts a large pretrained model to a new task by freezing its original weights and training a small pair of matrices alongside selected layers. Their product is a low-rank update that is added to the frozen weights, so only a tiny fraction of the parameters ever receive gradients. Introduced by Hu et al. at Microsoft in 2021, LoRA cut the trainable parameters of GPT-3 175B by 10,000 times and its training GPU memory by 3 times compared with full fine-tuning using Adam, while matching or beating full fine-tuning quality on RoBERTa, DeBERTa, GPT-2, and GPT-3. It is now widely used to customize large language models (LLMs), diffusion models, and vision transformers on modest hardware. LoRa, the long-range radio protocol for IoT devices, is an unrelated term.

How LoRA Works#

Fine-tuning changes a pretrained weight matrix W of size d × k by some update ΔW. LoRA assumes this update has a low intrinsic rank, so it can be written as the product of two thin matrices: ΔW = B × A, where A is r × k, B is d × r, and the rank r is far smaller than d or k. The layer's output becomes h = Wx + (α / r) × BAx, where α is a fixed scaling constant. A is initialized with small random values and B with zeros, so training starts from exactly the pretrained model. Only A and B are updated during backpropagation, which also removes the optimizer state for all the frozen weights.

For a 4096 × 4096 projection, full fine-tuning trains about 16.8 million weights, while a rank-8 adapter trains 8 × (4096 + 4096) = 65,536. The PyTorch script below builds such an adapter and merges it into the frozen layer:

import torch
from torch import nn

# A frozen pretrained 4096 x 4096 projection, like one in a transformer attention block
base = nn.Linear(4096, 4096).requires_grad_(False)

# LoRA's trainable rank-8 update: B @ A, scaled by alpha / r
# B starts at zero, so the adapted layer initially matches the pretrained one
r, alpha = 8, 16
A = nn.Parameter(torch.randn(r, 4096) * 0.01)
B = nn.Parameter(torch.zeros(4096, r))
trainable = A.numel() + B.numel()
total = trainable + sum(p.numel() for p in base.parameters())
print(f"Trainable: {trainable:,} of {total:,} parameters ({trainable / total:.2%})")

# After training, merge the update into the base weight so inference costs nothing extra
nn.init.normal_(B, std=0.01)  # stand-in for trained values
x = torch.randn(2, 4096)
adapted = base(x) + (alpha / r) * (x @ A.T @ B.T)
merged = nn.functional.linear(x, base.weight + (alpha / r) * B @ A, base.bias)
print(f"Merged layer matches adapter output: {torch.allclose(adapted, merged, atol=1e-5)}")

The adapter trains 65,536 of the layer's 16,846,848 parameters, 0.39%, and folding B × A into W gives the same outputs as running the adapter separately. Merged LoRA models therefore run exactly as fast as the original, an advantage over earlier adapter layers that add extra computation to every forward pass.

Using LoRA in Practice#

The original paper adapted only the attention query and value projections of the transformer, and many recipes now target every linear layer. The rank and α are the main hyperparameters: the Hugging Face PEFT library defaults to r = 8 and α = 8. Because the base model stays untouched, one deployment can keep a single copy of it and swap small task-specific adapters in and out. With rank 4 on the query and value projections of GPT-3 175B, each adapted checkpoint shrank from 350 GB to about 35 MB, roughly 10,000 times smaller.

• Language models: Teams adapt open LLMs to a domain, a style, or an instruction format on one GPU, often combining LoRA with instruction tuning data.
• Image generation: LoRA files teach Stable Diffusion and similar models a character, product, or art style, and are shared and stacked separately from the base checkpoint.
• Vision and multimodal models: Adapters specialize vision transformers and vision language models (VLMs) for medical, industrial, or document images without retraining the backbone.

Variants#

• QLoRA: Freezes the base model in 4-bit NormalFloat precision and trains LoRA adapters on top. Dettmers et al. used it to fine-tune a 65-billion-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning performance.
• DoRA: Splits each pretrained weight into a magnitude and a direction, and applies LoRA only to the direction, which narrows the accuracy gap to full fine-tuning at the same inference cost.

LoRA vs. Related Terms#

• Parameter-efficient fine-tuning: The umbrella term for methods that train only a small part of a model. Freezing layers is the simplest form, and Ultralytics YOLO exposes it as the freeze training argument, which keeps the first N layers of a YOLO26 model at their pretrained weights while the rest train.
• Full fine-tuning: Updates every weight and stores a full-size copy of the model per task. On programming and mathematics, full fine-tuning substantially outperformed standard LoRA, while LoRA suffered less catastrophic forgetting of the base model's other skills.
• Prompt tuning: Learns soft prompt embeddings that are prepended to the input while every model weight stays frozen.
• Transfer learning: The general idea of reusing knowledge from one task for another. LoRA is one efficient way to do it.
• Model merging: Combines the weights of separately fine-tuned models, and merging several LoRA adapters into one base model is a common case.