Late Chunking

Late chunking is a document-embedding technique that preserves surrounding context by encoding a long document before producing embeddings for its smaller sections. Instead of splitting text first and embedding every chunk independently, it uses a long-context model to create context-aware token representations, then pools the tokens belonging to each chunk into a separate vector. The resulting embeddings remain suitable for retrieval while carrying information from the broader document.

How Late Chunking Works#

A conventional retrieval pipeline normally follows this order: split, encode, store. This is efficient, but an isolated chunk may contain phrases such as “the component,” “this result,” or “it failed” without identifying what they refer to.

Late chunking changes the order:

1. The document is divided into tokens, while its desired chunk boundaries are recorded.
2. The complete token sequence, or the largest portion fitting the model’s context window, passes through a long-context transformer.
3. Through the model’s attention mechanism, each token representation incorporates information from other tokens in that context.
4. Token vectors are grouped according to the recorded boundaries and combined, commonly through mean pooling, to produce one embedding per chunk.

The key distinction is timing: boundaries still exist, but they are applied after contextual encoding rather than before it. The Jina AI explanation of late chunking provides a visual comparison of these two processing orders.

Why Context Preservation Matters#

Chunking helps documents fit model limits and lets search systems retrieve precise passages. Microsoft’s document chunking guidance notes that representing an entire large document with one vector can also compress too many ideas into a single representation.

However, smaller chunks increase the risk of context loss. Consider these adjacent passages:

• “The XR-12 pump is installed beside the cooling manifold.”
• “It must be replaced after 10,000 operating hours.”

Embedded independently, the second passage does not identify what “it” means. With late chunking, its token representations have already attended to “XR-12 pump,” making the passage more useful for a query such as “When should the XR-12 pump be replaced?”

Late chunking is especially relevant to retrieval-augmented generation, where selected passages provide grounding context to a language model. The broader Google Cloud RAG overview explains how retrieval connects embeddings, vector search, and grounded generation.

Related Chunking and Retrieval Methods#

Late chunking solves a different problem from several similarly named techniques:

• Semantic chunking chooses boundaries based on changes in meaning. Late chunking determines when contextual encoding occurs, so the two techniques can be combined.
• Contextual retrieval enriches chunks with additional explanatory text or metadata before indexing. Late chunking instead transfers context implicitly through token representations.
• Chunk overlap repeats text near neighboring boundaries. It can preserve local clues but increases index size and may create duplicate results. Cohere’s chunking strategy guide discusses how chunk size and overlap affect retrieval.
• Late interaction stores multiple token-level vectors and compares them during retrieval. Late chunking typically stores one pooled vector per chunk, making it compatible with a conventional vector database.

A reranker is also complementary rather than equivalent: it reorders retrieved candidates after the initial search, as illustrated by the Google Cloud ranking workflow.

Practical Implementation#

Some embedding APIs expose late chunking directly. The following request sends ordered passages from one document to the Jina Embedding API and returns one contextual embedding for each passage:

import os

import requests

chunks = [
    "The XR-12 pump is installed beside the cooling manifold.",
    "It must be replaced after 10,000 operating hours.",
]

payload = {
    "model": "jina-embeddings-v3",
    "task": "retrieval.passage",
    "late_chunking": True,
    "input": chunks,
}
headers = {"Authorization": f"Bearer {os.environ['JINA_API_KEY']}"}

response = requests.post(
    "https://api.jina.ai/v1/embeddings",
    headers=headers,
    json=payload,
    timeout=30,
)
response.raise_for_status()
print(len(response.json()["data"]))

The output count matches the number of input chunks, but each vector reflects the shared document context. An alternative self-managed workflow encodes the complete token sequence and pools the spans afterward, as demonstrated in Elastic’s late chunking implementation for vector search.

Real-World Applications and Guidance#

Two practical applications show where this context matters:

• Technical knowledge assistants: Product manuals often introduce a part once and later refer to it indirectly. Late chunking helps a support system retrieve the correct maintenance step rather than another passage containing similar generic language.
• Computer vision report search: A computer vision and RAG workflow may combine detections, inspection notes, captions, and maintenance records. After Ultralytics YOLO26 detects equipment or defects, late chunking can improve retrieval across the associated textual report. Visual records can also be explored using an image similarity search workflow.

Use late chunking when documents contain cross-references, pronouns, definitions, or narrative dependencies. Keep related chunks in document order, respect the embedding model’s context limit, retain headings and source metadata, and evaluate retrieval with realistic queries. It adds encoding cost because more tokens are processed together, so independent chunk embeddings may remain preferable for short, self-contained records.