Semantic Caching

Semantic caching stores the responses of a large language model (LLM) or other generative model and reuses them for new requests that mean the same thing as an earlier one, even when the wording differs. The cache turns each prompt into an embedding, finds the most similar stored prompt, and returns its saved answer when the similarity clears a threshold. A hit skips the model call entirely, so latency drops to milliseconds and the request consumes no tokens. Because matching works on meaning, "I forgot my password" and "How do I reset my password?" share one stored answer, which an exact-match cache keyed on the prompt text would have to store twice. Zilliz released the open-source GPTCache library for this pattern in 2023, and vector databases and API gateways now offer it as a built-in feature.

How a Semantic Cache Works#

A sentence embedding model encodes the incoming prompt, and a vector search compares it with the embeddings of previously answered prompts, usually by cosine similarity. Small caches fit in memory, and large ones live in a vector database with an approximate nearest-neighbor index. On a miss, the model generates a fresh answer, and the new prompt embedding and response are stored together, usually with an expiry time.

The cache key has to cover everything else that changes the correct answer. Responses generated under a different system prompt, model version, user permission level, or set of retrieved documents are stored under separate scopes, so one customer's cached answer is never served to another. Freshness needs the same care: when a price, policy, or source document changes, entries built on the old version keep matching new prompts, so each entry carries a time-to-live and the affected scope is cleared when its source data changes.

Choosing the Similarity Threshold#

The threshold decides how different two prompts can be and still share an answer. This example uses the all-MiniLM-L6-v2 model from Sentence Transformers as the encoder for a three-entry cache:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

# Earlier prompts whose answers are already stored in the cache
cached = ["How do I reset my password?", "What are your opening hours?", "How do I cancel my subscription?"]
cached_embeddings = model.encode(cached)

# Each new prompt reuses the closest cached answer if it clears the threshold
queries = ["I forgot my password, how can I change it?", "When does the store open?", "How do I cancel my order?"]
for query in queries:
    query_embedding = model.encode(query)
    scores = model.similarity(query_embedding, cached_embeddings)[0]
    best = scores.argmax().item()
    status = "hit" if scores[best] >= 0.7 else "miss"
    print(f"{query} -> {cached[best]} ({scores[best]:.2f}, {status})")

The password paraphrase scores 0.89 and is a correct hit. The opening-hours paraphrase scores only 0.61 and misses. "How do I cancel my order?" scores 0.71 against "How do I cancel my subscription?" and would return the wrong instructions. Embedding similarity measures shared wording and topic, so no single threshold separates these three cases. Production caches narrow the gap by fine-tuning the encoder on paraphrase pairs from their own traffic, setting thresholds per category, or passing each candidate hit to a reranker that scores the two prompts together before the cached answer is served.

Where Semantic Caching Fits#

Semantic caching pays off where many users ask the same questions and the answers change slowly: support chatbots, internal help desks, and retrieval-augmented generation (RAG) assistants over stable documentation. Personalized, time-sensitive, or creative requests gain little, since a reused answer is wrong or repetitive for them.

Two other caches in LLM serving work at different layers:

• Prompt caching: The model server stores the computed states of an exact repeated prompt prefix, which shortens prefill, and the model still generates a new answer.
• KV cache: Keys and values of earlier tokens are kept inside a single generation so each new token avoids recomputing them.

Semantic search uses the same embedding lookup to retrieve documents for a query, and a semantic cache uses it to retrieve finished answers, often inside an AI gateway that sits between applications and model providers.

Images can key a cache in the same way. embed() in Ultralytics YOLO returns a feature vector for each image, so a YOLO26 model trained for image classification can supply the lookup key that lets near-duplicate camera frames reuse a vision-language model's earlier description.