Model Quantization

Model quantization converts a neural network's weights, and often its activations, from 32-bit floating-point numbers (FP32) to lower-precision formats such as 16-bit floats (FP16), 8-bit integers (INT8), or 4-bit integers. Each value then takes fewer bits to store and move, and integer arithmetic runs faster and uses less energy on most processors. An INT8 model is a quarter the size of its FP32 original, which is what lets object detection models run in real time on phones, cameras, and other edge AI devices, and lets large language models (LLMs) fit on a single GPU.

How Quantization Works#

Quantization maps a range of real numbers onto a small set of integer levels. In the common affine scheme, each float x is stored as an integer q = round(x / scale) + zero_point, and it is recovered as x ≈ (q - zero_point) × scale. The scale sets the step size between levels, and the zero point makes sure the float value 0 maps to an exact integer. For INT8 there are 256 levels, so the range must be chosen carefully: a range too wide wastes levels on values that never occur, and a range too narrow clips outliers. Weights are known in advance, so their ranges come straight from the trained model, often computed separately for each output channel. Activations depend on the input, so their ranges are measured during calibration by running a small set of representative images or text samples through the model.

import numpy as np

rng = np.random.default_rng(0)
weights = rng.normal(0, 0.05, size=1000).astype(np.float32)

# Affine INT8 quantization maps the observed float range onto 256 integer levels
scale = (weights.max() - weights.min()) / 255
zero_point = np.round(-128 - weights.min() / scale)
q = np.clip(np.round(weights / scale + zero_point), -128, 127).astype(np.int8)

# Dequantize to measure the rounding error the model has to tolerate
restored = (q.astype(np.float32) - zero_point) * scale
print(f"Size: {weights.nbytes} -> {q.nbytes} bytes")
print(f"Max error: {np.abs(weights - restored).max():.5f} (half a step: {scale / 2:.5f})")

It prints Size: 4000 -> 1000 bytes and a maximum error of 0.00068, equal to half of one quantization step. The weights shrink fourfold, and no value moves by more than half a step from its original.

Post-Training Quantization vs. Quantization-Aware Training#

• Post-training quantization (PTQ): converts an already trained model, using a small calibration set to set activation ranges. It takes minutes and needs no training data beyond those samples. Dynamic quantization is a PTQ variant that computes activation ranges on the fly at inference, and weight-only quantization leaves activations in floating point.
• Quantization-aware training (QAT): simulates quantization during training or fine-tuning by rounding values in the forward pass, so the weights learn to tolerate the rounding. It recovers most of the accuracy PTQ loses, at the cost of extra training.

Teams usually start with PTQ and switch only when its accuracy drop is too large:

LLMs add their own methods. Weight-only 4-bit schemes such as GPTQ and AWQ compress billions of parameters while keeping activations in 16-bit, the GGUF format stores quantized models for local inference, and recent GPUs compute natively in low-precision floating-point formats such as FP8.

Quantization vs. Related Techniques#

• Mixed precision: a training technique that runs most operations in FP16 or BF16 while keeping a master copy of weights in FP32, to speed up training. Quantization targets inference and goes further, often to integers.
• Model pruning: removes weights or whole channels to reduce the number of operations, and it is often combined with quantization, which makes each remaining operation cheaper.
• Knowledge distillation: trains a small student model to imitate a large teacher. The student is frequently quantized afterward for deployment.

Quantization in Practice#

The speedup depends on hardware support: an INT8 model only runs faster on processors with INT8 kernels, such as NVIDIA GPUs with TensorRT, Intel CPUs with OpenVINO, and the neural processing units (NPUs) in phones. Ultralytics YOLO26 applies PTQ during export with quantize=8, calibrating on images from the data dataset, and model.train(quantize=8) fine-tunes a checkpoint with QAT instead. This example exports an INT8 model for LiteRT, Google's runtime for Android and embedded devices:

from ultralytics import YOLO

# Load a pretrained YOLO26 nano model
model = YOLO("yolo26n.pt")

# INT8 post-training quantization; images from coco8.yaml calibrate activation ranges
model.export(format="litert", quantize=8, data="coco8.yaml")

Measured on COCO with TensorRT, INT8 PTQ lowers YOLO26n's mAP50-95 only from 0.403 to 0.393, but costs YOLO26s more, from 0.479 to 0.441, and QAT brings YOLO26s back to 0.471.