FP8

FP8, or 8-bit floating point, is a low-precision numerical format that stores each value in eight bits. AI systems use FP8 to reduce memory traffic and accelerate matrix multiplications, convolutions, and other expensive operations on compatible hardware. Compared with FP16 or FP32, it represents fewer distinct values, so successful FP8 workflows combine fast low-precision computation with scaling and selective higher-precision operations to preserve model quality.

Link to this sectionHow FP8 Represents Numbers#

A floating-point value contains a sign, an exponent that controls range, and a mantissa that controls detail. FP8 commonly appears in two formats documented by PyTorch floating-point data types:

• E4M3: One sign bit, four exponent bits, and three mantissa bits. It provides more numerical detail but a narrower range.
• E5M2: One sign bit, five exponent bits, and two mantissa bits. It covers a wider range but rounds values more aggressively.

E4M3 is often suitable for weights and activations, while E5M2 can better accommodate gradients with large value ranges. Exact variants differ across platforms, as shown in the AMD low-precision floating-point documentation, so an FP8 model is not automatically portable between every accelerator and runtime.

Because eight bits cannot represent a tensor’s full original range and detail, frameworks usually multiply values by a scaling factor before conversion. NVIDIA’s FP8 scaling primer describes strategies such as delayed scaling, which derives future scales from previously observed maximum values. Scaling limits overflow, underflow, and saturation without requiring every operation to run in higher precision.

Link to this sectionFP8 vs Related Formats#

FP8 occupies a middle ground among common AI data types:

• FP16 or half precision: Uses twice as many bits, offering greater numerical detail and generally simpler training. FP8 can reduce storage and bandwidth further but requires more careful scaling.
• BF16: Keeps a wide exponent range while providing less fractional detail than FP16. It is often used as the higher-precision companion in FP8 training.
• INT8: Represents integers rather than floating-point values. INT8 model quantization normally relies on scales to map real values onto fixed integer levels, while FP8 retains an exponent for naturally nonuniform spacing.
• FP4: Uses only four bits and can provide greater compression, but its extremely limited range and granularity usually make accuracy preservation more difficult.

FP8 is frequently part of a mixed-precision workflow rather than the data type for every tensor. Accumulation, normalization, optimizer state, or sensitive layers may remain in BF16, FP16, or FP32. Also, numerical precision is different from the precision evaluation metric, which measures how many positive model predictions are correct.

Link to this sectionReal-World Applications#

Two practical applications illustrate why FP8 matters:

1. Large-model training: Transformer training repeatedly performs large matrix multiplications. An FP8-aware framework can cast eligible weights and activations to FP8 while retaining higher precision where stability requires it. This lowers bandwidth demand and can increase training throughput. The TorchAO quantized-training workflow includes tensorwise and rowwise scaling choices that trade maximum speed against better handling of outliers.

2. High-throughput vision inference: A data center may run object detection across hundreds of retail, traffic, or manufacturing video streams. FP8-capable kernels can reduce the time and memory used by eligible model layers, potentially lowering inference latency and increasing concurrent stream capacity. The TensorRT quantized-types guide explains how explicit quantize and dequantize operations describe FP8 execution.

Link to this sectionNumerical Risks and Hardware Support#

Poor scaling can cause large values to saturate and small values to round to zero. Outliers are especially problematic when one scale covers an entire tensor. These effects can shift confidence scores, destabilize training, or reduce detection accuracy, so developers should validate the converted model against its higher-precision baseline.

Native hardware support is equally important. NVIDIA Hopper architecture introduced FP8 Tensor Core acceleration, whereas the older NVIDIA A100 GPU supports FP16, BF16, and INT8 but not native FP8 computation. The TensorRT hardware support matrix should therefore be checked before selecting a deployment precision.

Link to this sectionWorking with FP8 in Practice#

This small PyTorch example demonstrates E4M3 conversion and the rounding error produced when FP8 values are restored to FP32:

import torch

values = torch.tensor([0.1, 1.0, 3.14, 100.0], dtype=torch.float32)

# Convert to E4M3 FP8, then restore the values for comparison.
fp8_values = values.to(torch.float8_e4m3fn)
restored = fp8_values.to(torch.float32)
absolute_error = (restored - values).abs()

print(torch.stack((values, restored, absolute_error), dim=1))

The documented Ultralytics TensorRT export workflow provides FP16 and calibrated INT8 options for Ultralytics YOLO rather than a one-argument FP8 export. FP8 deployment therefore requires a compatible downstream conversion toolchain. Whatever precision is selected, use Ultralytics benchmark mode on the target GPU to compare latency, throughput, memory use, and task accuracy before production deployment.