KV Cache
A KV cache stores the keys and values of past tokens so a transformer skips recomputing them, speeding up LLM generation. How it works and its memory cost.
A KV cache (key-value cache) stores the key and value vectors that a transformer's attention layers have already computed for earlier tokens, so they are not recomputed when the model generates the next token. Large language models (LLMs) produce text one token at a time, and each new token must attend to every token before it. With a KV cache, each generation step computes keys and values only for the newest token and reads the rest from memory, which makes long responses practical. The cost is memory: the cache grows with every token and often takes more GPU memory than the model weights in long-context or high-batch serving.
How the KV Cache Works#
In each attention layer, every token is projected into a query, a key, and a value. A token's query is compared against the keys of all tokens it can see, and the resulting weights mix their values into the output. In a decoder-only LLM, a token can only see itself and earlier tokens, so the keys and values of past tokens never change once computed. Generation therefore runs in two phases. In the prefill phase, the model processes the whole prompt in parallel and fills the cache. In the decode phase, it generates one token per step, adding that token's key and value to the cache, until it emits an end-of-sequence token or reaches a length limit:
Prefill is limited by compute, since it runs large matrix multiplications over many tokens at once. Decode is limited by memory bandwidth, since every step reads the whole cache and all model weights to produce a single token.
Why It Speeds Up Generation#
Without a cache, step t would recompute keys and values for all t tokens generated so far, so the total work grows quadratically with output length. With the cache, each step adds one key and one value per layer. The single-head example below decodes a 10-token sequence with a cache and checks the final output against a full recomputation:
import torch
torch.manual_seed(0)
d = 64
w_q, w_k, w_v = torch.randn(3, d, d) / d**0.5 # one attention head's projections
tokens = torch.randn(10, d) # embeddings of a 10-token sequence
# Decode token by token: project only the newest token and append it to the cache
k_cache, v_cache = torch.empty(0, d), torch.empty(0, d)
for x in tokens:
k_cache = torch.cat([k_cache, (x @ w_k)[None]])
v_cache = torch.cat([v_cache, (x @ w_v)[None]])
out_cached = torch.softmax((x @ w_q) @ k_cache.T / d**0.5, dim=-1) @ v_cache
# Without a cache, the last step re-projects keys and values for all 10 tokens
k_full, v_full = tokens @ w_k, tokens @ w_v
out_full = torch.softmax((tokens[-1] @ w_q) @ k_full.T / d**0.5, dim=-1) @ v_full
print(f"Same output: {torch.allclose(out_cached, out_full, atol=1e-5)}")
print(f"Key/value projections over 10 steps: cached {len(tokens)}, recomputed {sum(range(1, len(tokens) + 1))}")It prints Same output: True and cached 10, recomputed 55: the cache gives an identical result while projecting each token once, against 1 + 2 + ... + 10 projections if every step started over.
KV Cache vs. Related Concepts#
- Prompt caching: keeps the KV cache of a shared prompt prefix, such as a long system prompt or document, and reuses it across separate requests, so later requests skip that part of prefill.
- Model weights: the fixed parameters learned in training and shared by every request. A KV cache belongs to one sequence and is discarded when that sequence ends.
- Embeddings: the vectors that represent each token as it enters the model. Every layer derives its own keys and values from the token's hidden state, so a cache holds one key and one value per token per layer.
- Semantic caching: stores whole responses and returns them for similar later queries, skipping the model entirely.
KV Cache in Vision Models#
Vision language models (VLMs) turn an image into hundreds or thousands of visual tokens, and those tokens enter the KV cache like text, so a single high-resolution image can occupy as much cache as several pages of text. Streaming video models face the same growth with every frame. Meta's SAM 2, which runs in the Ultralytics Python package, applies a similar cache to video segmentation: its memory bank stores encoded features of earlier frames, and memory attention reads them as each new frame is segmented, keeping objects tracked through occlusions without reprocessing past frames.
Memory Cost and How to Reduce It#
The cache holds two vectors, a key and a value, per token for every layer and key-value head, so its size is 2 × layers × KV heads × head dimension × bytes per value × tokens. For Llama 2 7B in FP16, with 32 layers, 32 heads, and a head dimension of 128, that is 512 KiB per token, or 2 GiB for one 4,096-token sequence. Serving systems use several techniques to fit more sequences in memory:
- Grouped-query attention (GQA): several query heads share one key-value head. Llama 3 8B uses 8 KV heads for 32 query heads, cutting the cache to 128 KiB per token.
- PagedAttention: stores the cache in fixed-size blocks, like virtual memory pages, so memory is allocated as sequences grow. Introduced with the vLLM serving engine, it raised throughput 2–4× over earlier systems.
- KV cache quantization: stores keys and values in FP8 or INT8 instead of FP16, halving the cache.
- Sliding window attention: each layer attends only to the most recent tokens, which caps the cache at the window size.
- Multi-head latent attention (MLA): compresses keys and values into a small shared latent vector before caching. In DeepSeek-V2, it reduced the KV cache by 93.3% compared with the earlier DeepSeek 67B model.










