QLoRA

QLoRA (Quantized Low-Rank Adaptation) is a fine-tuning method that freezes a pretrained large language model (LLM) in 4-bit precision and trains small LoRA adapters on top of it, backpropagating through the quantized weights into the adapters. Introduced by Dettmers et al. in the 2023 QLoRA paper, it cut the GPU memory needed to fine-tune a 65-billion-parameter model from more than 780 GB to under 48 GB, enough for one workstation GPU, while matching the task performance of 16-bit fine-tuning. The paper's Guanaco models, fine-tuned this way, reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours of training on a single GPU.

How QLoRA Works#

QLoRA combines a frozen, compressed base model with three memory-saving techniques:

• 4-bit NormalFloat (NF4): Each weight is stored as a 4-bit code pointing to one of 16 levels placed at quantiles of a normal distribution, with one absolute-maximum scale per block of 64 weights. Pretrained model weights are roughly normally distributed, so quantile levels put more resolution near zero, where most weights sit.
• Double quantization: The per-block scales are themselves quantized to 8 bits in blocks of 256, which cuts their overhead from 0.5 to about 0.127 bits per parameter, a saving of roughly 3 GB on a 65B model.
• Paged optimizers: Optimizer states live in NVIDIA unified memory and move to CPU RAM during memory spikes, such as long sequences under gradient checkpointing, so training does not crash with out-of-memory errors.

In the forward pass, each frozen layer dequantizes its NF4 weights to bfloat16 just before its matrix multiplication and adds the output of the LoRA path. Backpropagation computes gradients through the dequantized weights but updates only the two low-rank LoRA matrices. The paper found that adapters on every linear layer of the transformer block were needed to match full fine-tuning:

The snippet below builds the NF4 levels the way the bitsandbytes library does and compares them with 16 evenly spaced levels on blocks of normally distributed weights:

import torch

torch.manual_seed(0)
normal = torch.distributions.Normal(0.0, 1.0)
# NF4: 16 levels at quantiles of a normal distribution, with an exact zero (as built in bitsandbytes)
positive = normal.icdf(torch.linspace(0.9677083, 0.5, 9)[:-1])
negative = -normal.icdf(torch.linspace(0.9677083, 0.5, 8)[:-1])
nf4 = torch.cat([negative, torch.zeros(1), positive]).sort().values
nf4 = nf4 / nf4.max()
int4 = torch.linspace(-1, 1, 16)  # evenly spaced 4-bit levels for comparison

weights = torch.randn(4096, 64) * 0.02  # pretrained weights are roughly normal, in blocks of 64
absmax = weights.abs().amax(dim=1, keepdim=True)  # one scale per block
for name, levels in (("NF4", nf4), ("INT4", int4)):
    codes = ((weights / absmax)[..., None] - levels).abs().argmin(dim=-1)  # 4-bit index per weight
    restored = levels[codes] * absmax
    print(f"{name} mean squared error: {(restored - weights).pow(2).mean():.3e}")

It prints a mean squared error of 3.395e-06 for NF4 and 4.053e-06 for evenly spaced levels: with the same 16 levels and the same block scales, NF4's quantile spacing cuts reconstruction error by about 16% on normally distributed weights. In practice, Hugging Face Transformers loads a model in NF4 through bitsandbytes with a BitsAndBytesConfig that sets load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, and bnb_4bit_compute_dtype=torch.bfloat16, and the PEFT library attaches the LoRA adapters.

QLoRA vs. Related Methods#

• LoRA: Trains the same kind of adapters on a base model kept in 16-bit precision. The 16-bit weights of a 65B model alone take about 130 GB, and NF4 stores them in roughly a quarter of that.
• Parameter-efficient fine-tuning (PEFT): The family of methods that train a small set of parameters, including LoRA, QLoRA, adapters, and prompt tuning.
• Post-training model quantization: Compresses a finished model for inference. A QLoRA-tuned model is often merged into 16-bit weights and then quantized again for serving.
• Quantization-aware training (QAT): Simulates low-precision rounding in the forward pass while all weights train, so the deployed model keeps its accuracy at low precision. Ultralytics YOLO26 runs INT8 QAT when trained with quantize=8, inserting fake quantization into its convolution and linear layers through NVIDIA ModelOpt so the learned weights survive INT8 export.

Trade-offs#

QLoRA trades speed for memory: dequantizing every frozen weight on each forward and backward pass makes a training step slower than with 16-bit LoRA. The adapters are small enough to store and swap per task, but merging them into the base model requires dequantizing it to 16 bits first. For the 65B model, the 4-bit weights and double-quantized scales take about 34 GB, leaving the rest of a 48 GB GPU for adapters, activations, and paged optimizer states.