License YOLO
Back to Ultralytics Glossary

KV Cache

A KV cache stores the keys and values of past tokens so a transformer skips recomputing them, speeding up LLM generation. How it works and its memory cost.

A KV cache (key-value cache) stores the key and value vectors that a transformer's attention layers have already computed for earlier tokens, so they are not recomputed when the model generates the next token. Large language models (LLMs) produce text one token at a time, and each new token must attend to every token before it. With a KV cache, each generation step computes keys and values only for the newest token and reads the rest from memory, which makes long responses practical. The cost is memory: the cache grows with every token and often takes more GPU memory than the model weights in long-context or high-batch serving.

How the KV Cache Works#

In each attention layer, every token is projected into a query, a key, and a value. A token's query is compared against the keys of all tokens it can see, and the resulting weights mix their values into the output. In a decoder-only LLM, a token can only see itself and earlier tokens, so the keys and values of past tokens never change once computed. Generation therefore runs in two phases. In the prefill phase, the model processes the whole prompt in parallel and fills the cache. In the decode phase, it generates one token per step, adding that token's key and value to the cache, until it emits an end-of-sequence token or reaches a length limit:

Prefill is limited by compute, since it runs large matrix multiplications over many tokens at once. Decode is limited by memory bandwidth, since every step reads the whole cache and all model weights to produce a single token.

Why It Speeds Up Generation#

Without a cache, step t would recompute keys and values for all t tokens generated so far, so the total work grows quadratically with output length. With the cache, each step adds one key and one value per layer. The single-head example below decodes a 10-token sequence with a cache and checks the final output against a full recomputation:

import torch

torch.manual_seed(0)
d = 64
w_q, w_k, w_v = torch.randn(3, d, d) / d**0.5  # one attention head's projections
tokens = torch.randn(10, d)  # embeddings of a 10-token sequence

# Decode token by token: project only the newest token and append it to the cache
k_cache, v_cache = torch.empty(0, d), torch.empty(0, d)
for x in tokens:
    k_cache = torch.cat([k_cache, (x @ w_k)[None]])
    v_cache = torch.cat([v_cache, (x @ w_v)[None]])
    out_cached = torch.softmax((x @ w_q) @ k_cache.T / d**0.5, dim=-1) @ v_cache

# Without a cache, the last step re-projects keys and values for all 10 tokens
k_full, v_full = tokens @ w_k, tokens @ w_v
out_full = torch.softmax((tokens[-1] @ w_q) @ k_full.T / d**0.5, dim=-1) @ v_full

print(f"Same output: {torch.allclose(out_cached, out_full, atol=1e-5)}")
print(f"Key/value projections over 10 steps: cached {len(tokens)}, recomputed {sum(range(1, len(tokens) + 1))}")

It prints Same output: True and cached 10, recomputed 55: the cache gives an identical result while projecting each token once, against 1 + 2 + ... + 10 projections if every step started over.

  • Prompt caching: keeps the KV cache of a shared prompt prefix, such as a long system prompt or document, and reuses it across separate requests, so later requests skip that part of prefill.
  • Model weights: the fixed parameters learned in training and shared by every request. A KV cache belongs to one sequence and is discarded when that sequence ends.
  • Embeddings: the vectors that represent each token as it enters the model. Every layer derives its own keys and values from the token's hidden state, so a cache holds one key and one value per token per layer.
  • Semantic caching: stores whole responses and returns them for similar later queries, skipping the model entirely.

KV Cache in Vision Models#

Vision language models (VLMs) turn an image into hundreds or thousands of visual tokens, and those tokens enter the KV cache like text, so a single high-resolution image can occupy as much cache as several pages of text. Streaming video models face the same growth with every frame. Meta's SAM 2, which runs in the Ultralytics Python package, applies a similar cache to video segmentation: its memory bank stores encoded features of earlier frames, and memory attention reads them as each new frame is segmented, keeping objects tracked through occlusions without reprocessing past frames.

Memory Cost and How to Reduce It#

The cache holds two vectors, a key and a value, per token for every layer and key-value head, so its size is 2 × layers × KV heads × head dimension × bytes per value × tokens. For Llama 2 7B in FP16, with 32 layers, 32 heads, and a head dimension of 128, that is 512 KiB per token, or 2 GiB for one 4,096-token sequence. Serving systems use several techniques to fit more sequences in memory:

  • Grouped-query attention (GQA): several query heads share one key-value head. Llama 3 8B uses 8 KV heads for 32 query heads, cutting the cache to 128 KiB per token.
  • PagedAttention: stores the cache in fixed-size blocks, like virtual memory pages, so memory is allocated as sequences grow. Introduced with the vLLM serving engine, it raised throughput 2–4× over earlier systems.
  • KV cache quantization: stores keys and values in FP8 or INT8 instead of FP16, halving the cache.
  • Sliding window attention: each layer attends only to the most recent tokens, which caps the cache at the window size.
  • Multi-head latent attention (MLA): compresses keys and values into a small shared latent vector before caching. In DeepSeek-V2, it reduced the KV cache by 93.3% compared with the earlier DeepSeek 67B model.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning