Late Chunking

Late chunking is a way of embedding document chunks in which a long-context embedding model encodes the whole document first, and the resulting token vectors are then split at the chunk boundaries and mean-pooled into one embedding per chunk. Every token has already seen the full document when it is pooled, so a chunk that says "it must be replaced" still carries what "it" refers to. Researchers at Jina AI introduced the method in the 2024 paper Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, which applies it to existing embedding models without extra training.

Why Chunk Embeddings Lose Context#

Retrieval-augmented generation (RAG) systems split documents into chunks so that each vector represents one focused passage and the retrieved text fits in a model's context window. Conventional chunking embeds each chunk on its own, which cuts every link to earlier text. Take two consecutive sentences from a manual: "The XR-12 pump circulates coolant through the press line." and "It must be replaced after 10,000 operating hours." Embedded alone, the second sentence never mentions the XR-12 pump, so a question about replacing that pump matches it only weakly in vector search, although it holds the answer.

How Late Chunking Works#

The document is tokenized in full, and the token span of each chunk is recorded; the boundaries can come from any splitter, including fixed-size or semantic chunking. The whole token sequence then passes through the transformer embedding model, where self-attention mixes information from every token into every other. Finally, the token vectors inside each span are averaged into that chunk's embedding. Documents longer than the model's input limit, such as the 8,192 tokens of Jina's jina-embeddings-v2 models, are encoded in overlapping macro chunks, which the paper calls long late chunking.

The example below embeds the pump sentence both ways with a small Sentence Transformers model and compares each version with the question:

import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
chunks = [
    "The XR-12 pump circulates coolant through the press line.",
    "It must be replaced after 10,000 operating hours.",
]
query = model.encode("When should the XR-12 pump be replaced?", convert_to_tensor=True)

# Conventional chunking: embed the second chunk on its own
naive = model.encode(chunks[1], convert_to_tensor=True)

# Late chunking: encode the whole document, then mean-pool only the second chunk's tokens
tokens = model.encode(" ".join(chunks), output_value="token_embeddings")
start = len(model.tokenizer(chunks[0])["input_ids"]) - 1  # tokens of [CLS] + chunk 1
late = tokens[start:-1].mean(dim=0)  # drop the final [SEP]

for name, emb in [("naive", naive), ("late", late)]:
    print(name, round(torch.cosine_similarity(query, emb, dim=0).item(), 3))

The independently embedded chunk scores 0.382 against the question, and the late-chunked version of the same sentence scores 0.62, because its tokens attended to "The XR-12 pump" before pooling.

Object detection made the same change in 2015. R-CNN ran a convolutional network separately on every cropped region proposal, and Fast R-CNN ran it once over the whole image and pooled each region from the shared feature maps, which made test-time detection 213 times faster with a VGG16 backbone. Ultralytics YOLO26 likewise runs its backbone once over the full image and predicts every box from those shared features, so no detection is computed from an isolated crop.

Late Chunking vs. Related Methods#

• Contextual retrieval: Prepends an LLM-written note to each chunk before indexing. It costs one LLM call per chunk and also helps keyword search, since the added context is text.
• ColBERT late interaction: Keeps one vector per token and compares them with the query at search time. Late chunking pools to a single vector per chunk, so the result fits any standard vector index.
• Chunk overlap: Repeats a few sentences across neighboring chunks. It preserves only nearby context and enlarges the index.
• Rerankers: Reorder retrieved candidates with a stronger model after the first search and combine well with late chunking.

When Late Chunking Helps#

The gains are largest when chunks depend on earlier text, as with pronouns, abbreviations defined once, and narratives such as reports and manuals, and smallest for short, self-contained records such as product titles. The method needs an embedding model that pools by averaging tokens and whose token-level outputs are accessible, and the whole document must be re-encoded whenever any part of it changes. Chunk vectors from late chunking go into an ordinary vector database unchanged. Jina's embedding API exposes the method as a late_chunking parameter that embeds a list of consecutive chunks as one document.