BFloat16 (BF16)

BFloat16 (brain floating point, BF16) is a 16-bit floating-point format with 1 sign bit, 8 exponent bits, and 7 fraction bits. It keeps the same exponent width as standard 32-bit floating point (FP32), so it covers the same range of values, from about 1.2 × 10⁻³⁸ to 3.4 × 10³⁸, while storing each number in half the memory. The cost is precision: 7 fraction bits give only two to three significant decimal digits. Deep learning tolerates that rounding well, and the format, conceived at Google Brain for Google's TPUs, has become the default 16-bit format for training large models on modern GPUs and accelerators.

How BF16 Represents Numbers#

A BF16 value is exactly the upper half of an FP32 value: the same sign and exponent, with the fraction cut from 23 bits to 7. Converting FP32 to BF16 therefore only rounds away the lower 16 bits, and converting back appends zeros, which makes the format cheap to implement in hardware. The other common 16-bit format, IEEE FP16 or half precision, splits its bits differently, with 5 exponent and 10 fraction bits, which buys precision at the expense of range. Comparing the three formats on the same values shows the trade:

import torch

# Largest value, smallest normal value, and spacing just above 1.0 for each format
for dtype in [torch.float32, torch.float16, torch.bfloat16]:
    info = torch.finfo(dtype)
    print(f"{dtype!s:15} max={info.max:.3g}  tiny={info.tiny:.3g}  eps={info.eps:.3g}")

# A large activation, a tiny gradient, and a value that needs fine precision
x = torch.tensor([70000.0, 1e-8, 1.001953125])
print("FP16:", x.half().tolist())
print("BF16:", x.bfloat16().tolist())

BF16 reaches a maximum of 3.39e38 against FP16's 65,504, and its smallest normal value, 1.18e-38, matches FP32. FP16 turns 70,000 into infinity and 1e-8 into zero, while BF16 keeps both as 70,144 and about 1.0e-8. FP16's finer spacing (eps 0.000977 against BF16's 0.00781) preserves 1.001953125 exactly, and BF16 rounds it to 1.0.

BF16 vs. FP16 for Training#

Range matters more than precision during training. Gradients often fall below FP16's smallest values and vanish, so FP16 mixed precision training multiplies the loss by a scale factor before backpropagation and lowers that factor whenever gradients overflow, a technique called loss scaling. BF16 needs no loss scaling, because its range equals FP32's, and BF16 training matches FP32 results across vision, speech, language, and recommendation models in the same number of iterations without changing hyperparameters. Mixed-precision training keeps a master copy of the weights in FP32 and runs matrix multiplications in BF16 with FP32 accumulation, since summing many 7-bit-fraction products in BF16 would lose too much. The choice between the 16-bit formats mostly follows the hardware:

PyTorch applies BF16 through automatic mixed precision with torch.autocast(device_type="cuda", dtype=torch.bfloat16). Ultralytics YOLO exposes the same switch for training with amp="bf16", which runs the forward pass under BF16 autocast and turns off the gradient scaler that its default FP16 mode uses, on CUDA GPUs with native BF16 support.

Hardware Support#

Google's second- and third-generation Cloud TPUs multiply in BF16 inside their matrix units and accumulate in FP32, and later TPU generations kept the format. NVIDIA GPUs run BF16 on Tensor Cores from the Ampere generation onward, including the A100, H100, and Blackwell GPUs, and Intel Xeon processors with Advanced Matrix Extensions (AMX) accelerate it on the CPU. On an NVIDIA A100, BF16 and FP16 Tensor Core math both peak at 312 teraFLOPS, 16 times the GPU's 19.5 teraFLOPS of standard FP32 math.

BF16 vs. Related Formats#

• Half precision (FP16): The IEEE 16-bit format, with more precision and far less range. It remains common for inference, where activations stay within its range and its extra precision helps.
• TF32: An NVIDIA Tensor Core math mode with BF16's 8-bit exponent and FP16's 10-bit fraction, which libraries can use for FP32 matrix multiplications on Ampere and newer GPUs while tensors stay in 32-bit storage.
• FP8: 8-bit floating-point formats such as E4M3 and E5M2, supported by Hopper and Blackwell GPUs, which halve memory again for training and inference of the largest models.
• Model quantization: Converts trained weights to integer formats such as INT8 or INT4 for deployment.

BF16 in Large Language Models#

Most open-weight large language models (LLMs) are trained in BF16 mixed precision and published with BF16 weights, which serve as the reference that quantized FP8 and INT4 versions are measured against. At 2 bytes per parameter, the weights of a 70-billion-parameter model take 140 GB in BF16, more than the 80 GB of a single NVIDIA H100, so serving such a model in BF16 requires splitting it across at least two GPUs.