FP4

FP4 is a family of 4-bit floating-point formats used to compress and accelerate AI models through model quantization. It typically represents each value with one sign bit, two exponent bits, and one mantissa bit, called E2M1. Compared with FP16, FP4 can greatly reduce memory traffic and storage, helping supported GPUs process large models faster. NVIDIA’s NVFP4 low-precision format is a prominent implementation designed for Blackwell-generation hardware. (developer.nvidia.com)

Link to this sectionHow FP4 Works#

With only 16 possible bit patterns, plain FP4 cannot accurately represent the wide range of values found in neural networks. Modern implementations therefore divide tensors into small blocks and store a shared scale for each block. NVIDIA’s NVFP4 quantization scheme, for example, uses blocks of 16 E2M1 values, while the open OCP Microscaling Formats specification defines MXFP4 with microscaling metadata.

During computation, an inference engine quantizes higher-precision values, performs supported matrix operations in FP4, and returns results in a higher precision when necessary. Current TorchAO NVFP4 inference support uses dynamic activation quantization and double scaling for weights and activations. (docs.nvidia.com)

Link to this sectionFP4 Compared With Related Formats#

• FP16 or half precision: Offers a much wider numerical range and is generally easier to deploy, but uses four times as many bits per value.
• FP8: Usually provides better accuracy and training stability than FP4, while requiring roughly twice the storage.
• BF16: Commonly used for training because its large exponent range reduces overflow risk. The 2025 FP4 All the Way study showed that carefully scaled FP4 training can approach BF16 performance.
• INT4: Uses integer levels rather than floating-point exponents. FP4 can represent values across different magnitudes more naturally, while INT4 may work well for weight-only compression.
• NVFP4 versus MXFP4: NVFP4 uses smaller blocks and higher-precision scales, typically improving accuracy at a modest metadata cost.

FP4 may also refer to an unrelated AMD FP4 processor socket. In AI, FP4 means four-bit floating-point arithmetic, not the older laptop CPU package.

Link to this sectionReal-World Applications#

1. Generative image inference: NVIDIA demonstrated FP4 image generation on RTX 50 Series GPUs, reducing memory use and accelerating diffusion workloads used in generative AI.

2. Large-model training and serving: Blackwell systems have used NVFP4 in MLPerf training to improve large language model throughput. FP4 can similarly benefit future memory-bound computer vision systems that require real-time inference. (developer.nvidia.com)

Link to this sectionUsing FP4 Effectively#

As of July 2026, hardware-accelerated FP4 matrix operations require NVIDIA Blackwell or newer GPUs; the H100 is Hopper-based and does not provide native FP4 acceleration. Developers should confirm compatibility through the TensorRT hardware support matrix, validate accuracy after conversion, and consider quantization-aware training using the TorchAO QAT workflow when post-training quantization causes excessive degradation. (docs.nvidia.com)

Ultralytics currently provides production-ready FP16 and INT8 export for YOLO26. This comparable workflow creates an optimized TensorRT engine:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
engine = model.export(format="engine", quantize=16)
results = YOLO(engine).predict("https://ultralytics.com/images/bus.jpg")

The Ultralytics TensorRT integration explains supported precision options, while YOLO benchmark mode helps compare accuracy and latency. For experimental FP4 deployment, follow TensorRT-RTX precision best practices and format-specific techniques such as Micro-Rotated GPTQ, since FP4 is not automatically more accurate or faster than a well-optimized INT4 or FP8 workflow.