Prompt Caching

Prompt caching, also called prefix caching, stores the intermediate results a large language model (LLM) computes for the beginning of a prompt and reuses them for later requests that start with exactly the same text. Many requests share a long, unchanging prefix, such as a system prompt, tool definitions, a set of examples, a long document, or the conversation so far, followed by a short new part. With a cache hit, the server processes only the new tokens, which shortens the time to the first output token and, with hosted APIs, cuts the price of the cached input to a fraction of the normal rate.

How Prompt Caching Works#

A transformer LLM answers in two phases. In prefill, every layer computes a key and a value vector for each prompt token; in decoding, each new token attends to all of those keys and values through the attention mechanism. Within one request, the KV cache keeps these tensors so each generated token does not recompute the prompt. Prompt caching keeps them across requests. Because attention in these models is causal, a token's keys and values depend only on the tokens before it, so two prompts with the same prefix have identical keys and values for that prefix, and the stored copy can stand in for recomputation.

The following example shows the reuse with GPT-2 from Hugging Face Transformers. It prefills a 409-token system prompt once, then answers two questions by running only their own tokens on top of the cached prefix, and compares the result with a full recomputation:

import copy

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

# A long system prompt shared by every request, followed by different user questions
system = "You answer questions about a camera store. " + "Returns are accepted within 30 days with a receipt. " * 40
questions = [" Question: Can I return a lens?", " Question: Do you ship abroad?"]
prefix_ids = tokenizer(system, return_tensors="pt").input_ids

with torch.no_grad():
    # Prefill the shared prefix once and keep its key-value cache
    prefix_cache = model(prefix_ids).past_key_values
    for question in questions:
        ids = tokenizer(question, return_tensors="pt").input_ids
        # Cache hit: only the question's tokens run through the model
        cached = model(ids, past_key_values=copy.deepcopy(prefix_cache)).logits[0, -1]
        # No cache: the whole prompt is computed from scratch
        full = model(torch.cat([prefix_ids, ids], dim=1)).logits[0, -1]
        same = torch.allclose(cached, full, atol=1e-3)
        print(f"Computed {ids.shape[1]} of {prefix_ids.shape[1] + ids.shape[1]} tokens, same next-token scores: {same}")

It prints Computed 8 of 417 tokens, same next-token scores: True and Computed 7 of 416 tokens, same next-token scores: True: each request ran under 2% of its prompt through the model and got the same next-token scores as the full computation.

Open-source model serving engines apply this automatically. vLLM keeps the KV cache in fixed-size blocks, the memory layout of PagedAttention, reuses the blocks of the longest matching prefix for each new request, and evicts the least recently used blocks when GPU memory runs low. SGLang's RadixAttention keeps cached prefixes in a radix tree so that requests sharing any leading portion reuse it, and the SGLang paper reports up to 6.4x higher throughput than other state-of-the-art inference engines. The Prompt Cache paper by In Gim and colleagues, presented at MLSys 2024, extends reuse beyond exact prefixes to declared prompt modules such as templates and documents, and reported time-to-first-token reductions ranging from 8x on GPU to 60x on CPU.

Prompt Caching in LLM APIs#

Hosted APIs expose the same mechanism with a price on it. OpenAI's prompt caching is on by default for supported models, while Anthropic's Claude prompt caching is enabled with a cache_control field, either once per request or on up to four content blocks. Anthropic bills a cache read at a tenth of the base input price or less and a write to its default five-minute cache at 1.25 times the base price, and each hit resets the five-minute lifetime. Both providers match only an identical prefix: Anthropic compares tools, then the system prompt, then messages, so editing a tool definition invalidates everything cached after it, and OpenAI's guide places stable instructions and reference material first and per-user details last.

Workloads with long shared prefixes gain the most:

• AI agents: Each step resends the tool definitions and the whole history, which is the previous step's prompt plus the latest action and its result.
• Chatbots: Every turn of a conversation extends the previous turn's prompt, so the history is cached up to the new message.
• Document question answering: Many questions about one long contract, report, or repository reuse the processed document, and retrieval pipelines reuse it whenever the same passages are retrieved again.
• Few-shot prompts: Dozens of fixed examples are computed once and reused for every input.

Open-vocabulary vision models reuse prompt computation in a similar way. Ultralytics YOLOE encodes class prompts such as "forklift" with a MobileCLIP text encoder once, when set_classes() is called, and stores the resulting embeddings in the model, so later predictions skip the text encoder entirely; save_prompt_embeddings() writes them to an NPZ file that a later session loads with load_prompt_embeddings() instead of running the encoder again.

Prompt Caching vs. Related Terms#

• KV cache: Reuses keys and values across the tokens of one generation. Prompt caching reuses them across separate requests.
• Semantic caching: Stores whole responses and returns one when a new query means the same thing as an earlier one, skipping the model entirely. Prompt caching still runs the model on the new part of the prompt and generates a fresh answer.
• Prompt compression: Shortens the prompt itself, which suits text that changes with every request. The two combine well, with the fixed part of a prompt cached and the variable part compressed.
• Prompt engineering: Decides what the prompt says. Caching changes only how much of it the server recomputes, and a hit leaves the next-token scores unchanged up to floating-point rounding.