GGUF
GGUF is the single-file format llama.cpp uses to store an LLM's weights, tokenizer, and metadata, often quantized to 4-8 bits for local hardware.
GGUF is a binary file format for storing models that run on GGML, the C tensor library behind llama.cpp, and on the tools built on it. One GGUF file holds everything needed to run a large language model (LLM): the architecture and hyperparameters, the tokenizer vocabulary, the chat template, and the model weights, usually compressed by model quantization to between 2 and 8 bits per weight. The llama.cpp project introduced GGUF in August 2023 to replace its earlier GGML, GGMF, and GGJT formats, and it has become the standard way to share open-weight models for local inference on a laptop CPU, an Apple Silicon Mac, or a consumer GPU, with thousands of GGUF repositories on Hugging Face.
Inside a GGUF File#
The GGUF specification defines four parts in order. A header holds the magic bytes GGUF, the format version (currently 3), and the counts of tensors and metadata entries. Typed key-value metadata follows, such as general.architecture, llama.context_length, tokenizer.ggml.tokens, and tokenizer.chat_template, which embeds the model's tokenization and prompt format in the file itself. A table then lists each tensor's name, shape, data type, and offset, and the tensor data comes last, padded to an alignment of 32 bytes by default so the runtime can memory-map the file and load it almost instantly. Readers look up metadata by key, so new architectures and fields can be added without breaking older files.
Each tensor carries its own type, so one file can mix precisions. The gguf Python package, maintained in the llama.cpp repository, writes and reads the format and implements several quantization types:
import numpy as np
from gguf import GGMLQuantizationType, GGUFReader, GGUFWriter
from gguf.quants import dequantize, quantize
weights = np.random.default_rng(0).standard_normal((1024, 1024)).astype(np.float32)
# A GGUF file holds metadata key-value pairs plus named tensors, each with its own type
writer = GGUFWriter("tiny.gguf", arch="llama")
writer.add_context_length(4096)
for qtype in [GGMLQuantizationType.F16, GGMLQuantizationType.Q8_0, GGMLQuantizationType.Q4_0]:
writer.add_tensor(f"weight.{qtype.name}", quantize(weights, qtype), raw_dtype=qtype)
writer.write_header_to_file()
writer.write_kv_data_to_file()
writer.write_tensors_to_file()
writer.close()
# Read the file back: metadata keys, then each tensor's bits per weight and rounding error
reader = GGUFReader("tiny.gguf")
print([name for name in reader.fields if not name.startswith("GGUF.")])
for tensor in reader.tensors:
error = np.abs(dequantize(tensor.data, tensor.tensor_type).reshape(weights.shape) - weights).mean()
print(f"{tensor.tensor_type.name}: {8 * tensor.n_bytes / weights.size:.1f} bits per weight, mean error {error:.4f}")It lists the two metadata keys, general.architecture and llama.context_length, then reports 16.0 bits per weight with a mean error of 0.0001 for F16, 8.5 bits and 0.0045 for Q8_0, and 4.5 bits and 0.0719 for Q4_0. Q8_0 and Q4_0 store blocks of 32 weights with one FP16 scale per block, which is where the extra half bit comes from.
Quantization Types#
GGUF's quantization types fall into three generations. The original block formats such as Q4_0 and Q8_0 use one scale per 32 weights. K-quants such as Q4_K and Q6_K group blocks into super-blocks of 256 weights with quantized scales, and file names such as Q4_K_M mark a mix that keeps the most sensitive tensors at higher precision. I-quants such as IQ2_XXS and IQ4_XS reach about 2 to 4 bits per weight, usually guided by an importance matrix that llama-imatrix computes from sample text so the most influential weights keep the most precision. Newer additions include MXFP4, the 4-bit floating-point format OpenAI used to release gpt-oss. Most published GGUF files come from the same conversion path:
GGUF vs. Other Model Formats#
- GGML, GGMF, and GGJT: The earlier llama.cpp formats stored hyperparameters as an untyped list with no architecture field, so adding a parameter broke existing files.
- Safetensors: Hugging Face's format for safely storing raw tensors, usually in full or half precision, with the configuration and tokenizer kept in separate files.
- ONNX: Stores a full computation graph along with the weights, so any ONNX-compatible runtime can execute the model without architecture-specific code.
Running GGUF Models#
llama.cpp runs GGUF files from the command line or as llama-server, which exposes an OpenAI-compatible API, and it can offload any number of layers to a GPU while the rest run on the CPU, which lets a model larger than GPU memory still run. Ollama and LM Studio run GGUF models behind a model library and a chat interface, and the Hugging Face Hub shows each GGUF file's metadata and tensor types in the browser. Local inference keeps prompts on the machine, which matters for data privacy and offline edge AI uses. The Ultralytics LLM interface can call a GGUF model served by llama-server or Ollama through their OpenAI-compatible Chat Completions endpoints, so a YOLO26 pipeline can ask a local model to describe its detections without sending data to a cloud API. The format also reaches beyond language models: the ComfyUI-GGUF custom nodes load quantized diffusion models such as FLUX.1 from GGUF files so they fit in the memory of consumer GPUs.










