FP4

FP4 is a 4-bit floating-point number format used to store and compute the weights and activations of neural networks in model quantization. AI hardware and software use the E2M1 layout: one sign bit, two exponent bits, and one mantissa bit, which can represent only 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, and ±6. Each value takes half the memory of FP8 and a quarter of FP16 (half precision), which matters most for large language models and other generative AI models whose inference latency is dominated by moving weights out of memory. Native FP4 matrix math arrived with NVIDIA's Blackwell GPUs and AMD's Instinct MI350 series, and FP4 is used for both inference and, increasingly, training.

How FP4 Works#

Fifteen distinct values cannot cover the range of real weights and activations, so every practical FP4 format splits a tensor into small blocks and stores one shared scale per block, chosen so the block's largest magnitude maps near the E2M1 maximum of 6. The two main formats differ in block size and scale precision:

• MXFP4: Defined in the Open Compute Project's Microscaling (MX) specification, MXFP4 groups 32 values under one 8-bit power-of-two scale (E8M0). As an open specification, it is supported across vendors, including AMD's MI350 series and NVIDIA's Blackwell GPUs.
• NVFP4: NVIDIA's format groups 16 values under one FP8 (E4M3) scale and adds a second FP32 scale per tensor. The smaller blocks and finer scales reduce quantization error, and NVIDIA reported 1% or less accuracy loss versus FP8 on DeepSeek-R1-0528 benchmarks.

The snippet below rounds a tensor with a few outliers to the E2M1 values using one scale for the whole tensor, then one scale per 32 values and per 16 values, the MXFP4 and NVFP4 block sizes:

import torch

# The eight non-negative values FP4 E2M1 can represent; the sign bit mirrors them
grid = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])

torch.manual_seed(0)
x = torch.randn(4096)
x[::512] *= 20  # a few large outliers, as real weight and activation tensors have

# One shared scale for the whole tensor, then MXFP4-sized and NVFP4-sized blocks
for block in [4096, 32, 16]:
    blocks = x.reshape(-1, block)
    scale = blocks.abs().amax(dim=1, keepdim=True) / 6.0  # map each block's largest value to 6
    nearest = ((blocks.abs() / scale).unsqueeze(-1) - grid).abs().argmin(dim=-1)
    dequantized = grid[nearest] * blocks.sign() * scale
    error = (dequantized.flatten() - x).abs().mean()
    print(f"Block size {block:>4}: mean absolute error {error:.4f}")

With one scale for the whole tensor, the outliers stretch the scale so far that most ordinary values round to zero, giving a mean absolute error of 0.7238. Blocks of 32 and 16 values confine each outlier to a single block, so every other block keeps a fine scale, and the error drops to 0.0901 and 0.0747.

FP4 Inference and Training#

For inference, an inference engine such as TensorRT quantizes weights, and often activations, to FP4 after training, runs the matrix multiplications in FP4, and accumulates results at higher precision. TorchAO's NVFP4 and MXFP4 configurations bring the same workflow to PyTorch on supported GPUs. OpenAI released gpt-oss-120b with its mixture-of-experts weights post-trained in MXFP4, which lets the 117-billion-parameter model run on a single 80 GB GPU.

Training in FP4 extends mixed precision training down to 4 bits and is harder than inference, because gradients and activations change range throughout training. NVIDIA's NVFP4 pretraining study trained a 12-billion-parameter model on 10 trillion tokens in NVFP4, with training loss and downstream accuracy comparable to an FP8 baseline, by combining random Hadamard transforms to spread out outliers, stochastic rounding for gradients, and a few layers kept at higher precision.

Deploying FP4 Models#

Native FP4 tensor cores require NVIDIA Blackwell GPUs or later, such as the B200 and the GeForce RTX 50 series, or AMD Instinct MI350 accelerators for MXFP4; Hopper GPUs such as the H100 accelerate FP8 but not FP4. The TensorRT support matrix lists which precisions each GPU supports. Accuracy must be checked after conversion and before model deployment, since sensitive layers or small models can degrade, and quantization-aware training recovers accuracy when post-training quantization is not enough. Ultralytics YOLO26 TensorRT export builds engines at FP16 with quantize=16 or INT8 with quantize=8, the low-precision options it offers for YOLO models.

FP4 vs. Related Formats#

• FP8: FP8 (E4M3 or E5M2) has far more distinct values and a wider range, and because FP8 training is widely adopted for large models, FP8 is the usual baseline that FP4 results are measured against.
• BFloat16: BF16 remains the standard format for training and serving large models at full quality, with four times the bits of FP4.
• INT4: INT4 spaces its 16 levels evenly, while FP4 packs more levels near zero, where most weights and activations sit. INT4 remains common for weight-only compression of language models with methods such as GPTQ and AWQ.
• AMD Socket FP4: FP4 is also the name of an unrelated AMD laptop processor socket, the package used by Carrizo-generation mobile APUs such as the FX-8800P.