Contextual Retrieval

Contextual retrieval is an indexing technique for retrieval-augmented generation (RAG) in which a large language model (LLM) reads each whole document and writes a short note, usually 50 to 100 tokens, that situates one chunk within it. The note is prepended to the chunk before the chunk is embedded and added to a keyword index, so search can match information the chunk alone never states. Anthropic introduced contextual retrieval in September 2024.

The Problem It Solves#

RAG pipelines split documents into chunks small enough to embed precisely and to fit several into a model's context window, and a chunk cut out of its document often loses the facts that make it findable. Anthropic's example is a chunk from a financial filing that reads "The company's revenue grew by 3% over the previous quarter." It names neither the company nor the quarter, so a question about ACME's revenue in Q2 2023 has nothing to match in it, for embeddings or for keyword search. Contextual retrieval stores it as "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."

Object crops lose context the same way text chunks do. The object cropping solution in Ultralytics runs YOLO26 on images or video and saves every detection as its own file, crop_1.jpg, crop_2.jpg, and so on, so a crop shows a truck but not which camera, site, or scene it came from. Indexing each crop with a short prefix describing its source frame, such as the camera name, the time, and the other objects detected around it, restores that context for keyword and semantic search.

How Contextual Retrieval Works#

For every chunk, the LLM receives the full document and the chunk and is asked to "give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk." Anthropic used Claude 3 Haiku with prompt caching, which processes each document once and reuses it for all of its chunks, and put the one-time cost at $1.02 per million document tokens. The contextualized chunks go into an embedding index ("contextual embeddings") and a BM25 keyword index ("contextual BM25"). At query time both indexes are searched and their results merged, as in hybrid search, then a reranker scores the top 150 candidates and the best 20 chunks go into the prompt.

Embedding the example chunk with and without its context shows the effect on vector search:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Anthropic's example chunk and the context an LLM wrote for it from the full filing
chunk = "The company's revenue grew by 3% over the previous quarter."
context = "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million."
other = "Globex Corporation reported that Q2 2023 revenue fell 2% on weaker demand."

query = "How did ACME's revenue change in Q2 2023?"
candidates = {"plain chunk": chunk, "contextual chunk": f"{context} {chunk}", "Globex chunk": other}

# Cosine similarity of the query to each indexed text
scores = model.similarity(model.encode(query), model.encode(list(candidates.values())))[0]
for name, score in zip(candidates, scores):
    print(f"{name}: {score:.2f}")

The plain chunk scores 0.49 against the question, the same as a chunk about a different company. With its context prepended, the chunk scores 0.83 and ranks first by a wide margin.

Contextual Retrieval vs. Related Techniques#

• Late chunking: Gives each chunk's embedding document context through the embedding model's attention, with no LLM call per chunk. The stored chunk text stays unchanged, so keyword search gains nothing.
• Semantic chunking: Chooses where chunks begin and end. It combines with contextual retrieval, which fills in what each chunk leaves unsaid.
• Title and heading prefixes: The cheaper, older practice of prepending the same document title or section heading to every chunk. Contextual retrieval writes a different note for each chunk.
• Context engineering: The broader practice of deciding what goes into a model's context window at inference time. Contextual retrieval works one step earlier, on what gets indexed.

Measured Gains#

Anthropic measured retrieval failure as 1 minus recall@20, the share of queries whose relevant chunk was missing from the top 20 results. The baseline pipeline failed on 5.7% of queries. Contextual embeddings lowered that to 3.7%, a 35% reduction, and adding contextual BM25 lowered it to 2.9%, a 49% reduction. Reranking the combined results brought the failure rate to 1.9%, a 67% reduction from the baseline.