Semantic Chunking

Semantic chunking splits documents into chunks for retrieval-augmented generation (RAG) and search at the points where the topic changes, found by comparing the embeddings of neighboring sentences, instead of every fixed number of characters or tokens. Each chunk then covers one idea, so its single vector in the vector database represents that idea well, and a retrieved chunk hands the large language model (LLM) a complete thought instead of the end of one section and the start of the next.

How Semantic Chunking Works#

The common algorithm, popularized by Greg Kamradt's text-splitting tutorials and implemented as the SemanticChunker in LangChain, has four steps. The text is split into sentences, and each sentence, often grouped with its neighbors to smooth out noise, is embedded. The cosine similarity between each pair of consecutive sentences is turned into a distance. A threshold marks breakpoints where the distance is unusually large: LangChain defaults to the 95th percentile of all distances in the document and also offers standard-deviation, interquartile, and gradient thresholds. Sentences between breakpoints form a chunk. The semantic splitter in LlamaIndex uses the same percentile breakpoint.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
sentences = [
    "The XR-12 coolant pump circulates water through the press line.",
    "Inspect the XR-12 pump seals for leaks every week.",
    "A worn pump bearing grinds loudly before it fails.",
    "The staff cafeteria now opens for breakfast at 7:00.",
    "The cafeteria menu adds vegetarian lunches in May.",
]

# Cosine distance between each sentence and the next one
embeddings = model.encode(sentences, normalize_embeddings=True)
distances = 1 - np.sum(embeddings[:-1] * embeddings[1:], axis=1)

# Start a new chunk where the distance exceeds the 95th percentile
threshold = np.percentile(distances, 95)
chunks = [[sentences[0]]]
for sentence, distance in zip(sentences[1:], distances):
    if distance > threshold:
        chunks.append([])
    chunks[-1].append(sentence)
print(np.round(distances, 2))
print(*[" ".join(chunk) for chunk in chunks], sep="\n")

The distances between neighbors are 0.56, 0.64, 0.98, and 0.62, so the only breakpoint falls at the 0.98 jump from the pump bearing to the cafeteria, and the output is one three-sentence chunk about the XR-12 pump and one two-sentence chunk about the cafeteria.

Chunking Strategies Compared#

• Fixed-size chunking: Splits every N tokens, usually with some overlap between neighbors. It is fast and predictable and ignores topic boundaries.
• Recursive splitting: Splits on paragraph breaks, then line breaks, then sentences, until each piece fits a size limit. LangChain recommends its recursive character splitter for generic text.
• Structure-aware splitting: Follows headings, Markdown sections, HTML elements, or the page layout of a PDF.
• Semantic chunking: Follows changes in meaning, at the cost of embedding every sentence during indexing and of chunk sizes that vary widely, which is why implementations add minimum and maximum lengths.
• Late chunking and contextual retrieval: Address a chunk that lacks context from the rest of its document, such as a pronoun whose referent sits in an earlier chunk, and combine with any of the boundary methods above.

Action chunking in robotics shares only the name: there, a policy predicts a sequence of future motor commands as one unit.

Semantic Chunking Beyond Text#

Transcripts from speech-to-text systems have no paragraph breaks, so embedding-based breakpoints are a common way to find where a meeting or podcast changes topic. Video can be chunked by content too, so that each clip shows one event for video understanding or multimodal RAG instead of an arbitrary 10-second slice. Ultralytics YOLO26 supplies such boundaries for scenes with people and vehicles: tracking mode gives each detected object a persistent ID across frames, so a pipeline can open a clip when a new ID enters the scene and close it when every tracked ID has left.

Does Semantic Chunking Improve Retrieval?#

Topic-based boundaries matter most when unrelated subjects sit next to each other without headings, as in transcripts, scraped web pages, and concatenated notes. Semantic chunks also depend on the embedding model and threshold, so the same document chunks differently after either changes, and both should be tuned against recall on real queries. The study Is Semantic Chunking Worth the Computational Cost? (2024) compared semantic and fixed-size chunking on document retrieval, evidence retrieval, and retrieval-based answer generation, and found that the extra computation of semantic chunking did not produce consistent gains over fixed-size chunks.