Semantic Search

Semantic search ranks documents, images, or other items by how closely their meaning matches the meaning of a query, instead of by the words they share with it. A search for "my cat is hungry" can return a page titled "Feeding schedule for a new kitten" although the two share no words. Modern systems do this with embeddings: a neural encoder maps queries and items to vectors that sit close together when their meanings are similar, and vector search finds the nearest items. Web search moved in this direction in 2019, when Google began applying BERT to understand search queries, a change it said affected one in ten English-language searches in the US.

How Semantic Search Works#

Most semantic search systems retrieve with a bi-encoder and refine with a reranker. The bi-encoder embeds every document once, ahead of time, and the vectors go into a vector database. At query time only the query is embedded, and an approximate nearest-neighbor index returns the top candidates by cosine similarity. Sentence-BERT (2019) made this practical for sentences: finding the most similar pair among 10,000 sentences took about 65 hours with a BERT cross-encoder and about 5 seconds with its sentence embeddings. Dense Passage Retrieval (2020) then showed that learned embeddings beat BM25 keyword ranking by 9 to 19 points of top-20 passage retrieval accuracy on open-domain question answering. A cross-encoder reranker finally reads the query and each candidate together to reorder the short list more precisely.

This example compares word overlap with embedding similarity using the Sentence Transformers library:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

documents = [
    "How to reset a forgotten account password",
    "Feeding schedule for a new kitten",
    "Quarterly sales report for the retail division",
    "Best hiking trails near Denver",
]
query = "my cat is hungry"

# Keyword search: documents sharing at least one word with the query
query_words = set(query.lower().split())
print("Keyword matches:", [d for d in documents if query_words & set(d.lower().split())])

# Semantic search: rank documents by cosine similarity of their embeddings
doc_embeddings = model.encode(documents)
query_embedding = model.encode(query)
scores = model.similarity(query_embedding, doc_embeddings)[0]
best = int(scores.argmax())
print(f"Semantic match: {documents[best]!r} (cosine {scores[best]:.2f})")

Keyword matching returns an empty list, because no document contains "my," "cat," "is," or "hungry." Semantic search returns the kitten feeding schedule with a cosine similarity of 0.56, the closest document in meaning.

Semantic Search vs. Related Terms#

• Keyword search: Ranks documents by exact term matches with formulas such as BM25. It handles names, part numbers, and rare words that embeddings can blur, and misses synonyms and paraphrases.
• Hybrid search: Runs keyword and semantic retrieval together and merges the results, a common setup in production search and RAG systems.
• Semantic segmentation: An unrelated computer vision task that labels every pixel of an image with a class and shares only the word "semantic."
• Semantic caching: Applies the same meaning-based matching to LLM requests, reusing a stored answer when a new prompt means the same as an earlier one.

Real-World Applications#

• Retrieval-augmented generation (RAG): Finds the passages a language model needs, even when the user's wording differs from the documentation's.
• Patent search: Surfaces prior art that describes the same invention in different terminology.
• Retail product search: Matches a query such as "shoes for a rainy commute" with waterproof sneakers whose listings never use those words.
• Recommendation systems: Suggest articles or products whose embeddings sit near those a user has already chosen.

Searching Images by Meaning#

Models trained with contrastive learning on image-caption pairs, such as CLIP, embed text and images into one space, so a text query can rank photos that carry no tags or captions. The Ultralytics semantic image search solution is built this way: its VisualAISearch class embeds every image in a folder with CLIP once, embeds each text query into the same space, and returns up to 30 images whose cosine similarity to the query passes a threshold, ranked from most to least similar. A query such as "a dog sitting on a bench" then finds matching photos in an unlabeled collection.