Vector Search

Vector search retrieves the items whose embeddings lie closest to a query embedding under a similarity measure such as cosine similarity, dot product, or Euclidean distance. An embedding model places items with similar meaning or appearance near each other, so the nearest vectors are the most similar documents, images, or products, even when they share no keywords with the query. Exact vector search is a k-nearest neighbors (k-NN) lookup over every stored vector. At scale, systems use approximate nearest neighbor (ANN) indexes that compare the query with a small fraction of the vectors and still return nearly all of the true neighbors.

How Vector Search Works#

The query and the stored items are embedded by the same model or by a pair trained together: a sentence encoder for text, CLIP for matching text to images, or a vision backbone for matching images to images. The metric should match how the model was trained. For unit-length vectors, cosine similarity equals the dot product, and ranking by Euclidean distance gives the same order. Exact search costs one comparison per stored vector for every query, which is fast for tens of thousands of vectors and too slow for hundreds of millions, so large systems use one of four families of ANN index:

• Graph indexes such as HNSW: Link each vector to its near neighbors in a layered graph and answer a query by walking greedily toward it.
• Clustering indexes (IVF): Group the vectors with k-means and scan only the clusters whose centroids are closest to the query.
• Vector quantization: Compresses each vector to a few bytes, as in product quantization, so distances are approximated from compact codes held in memory.
• Disk-based graphs: Keep a graph index on SSD, as DiskANN does, to search billions of vectors on a single machine.

The core idea of an IVF index fits in a short NumPy and scikit-learn example that compares it with exact search:

import numpy as np
from sklearn.cluster import KMeans

# 20,000 embeddings grouped by topic, normalized so a dot product is cosine similarity
rng = np.random.default_rng(0)
topics = rng.normal(size=(200, 64))
vectors = topics[rng.integers(200, size=20000)] + 0.5 * rng.normal(size=(20000, 64))
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
query = vectors[0] + 0.1 * rng.normal(size=64)

# Exact search: score every stored vector
exact = np.argsort(-(vectors @ query))[:10]

# IVF index: cluster the vectors, then scan only the lists of the 5 closest centroids
kmeans = KMeans(n_clusters=100, n_init=1, random_state=0).fit(vectors)
probed = np.argsort(-(kmeans.cluster_centers_ @ query))[:5]
candidates = np.flatnonzero(np.isin(kmeans.labels_, probed))
approx = candidates[np.argsort(-(vectors[candidates] @ query))[:10]]

print(f"scanned {len(candidates) / len(vectors):.1%} of vectors")
print(f"recall@10: {len(set(exact) & set(approx)) / 10:.0%}")

Probing 5 of the 100 clusters scans 5.1% of the vectors and still finds all 10 true nearest neighbors, a recall@10 of 100%. Probing more clusters raises recall and cost together, which makes the number of probes the main setting to tune in an IVF index.

Vector Search vs. Related Terms#

• Keyword search: Looks up exact terms in an inverted index and ranks them with formulas such as BM25. It excels at names, codes, and rare words and misses synonyms.
• Semantic search: Search by meaning, the goal that vector search most often serves. Semantic search systems often add query rewriting or a reranker on top.
• Hybrid search: Runs keyword and vector search together and merges the two ranked lists.
• Vector database: The system that stores vectors with metadata and serves vector search with filtering, updates, and persistence.

Real-World Applications#

• Retrieval-augmented generation (RAG): Finds the document passages a language model needs to answer a question.
• Object re-identification: Matches the appearance embedding of a person or vehicle against earlier sightings to link the same object across cameras.
• Facial recognition: Compares a face embedding with enrolled faces and accepts the nearest one when it is close enough.
• Recommendation systems: Suggest products, songs, or videos whose embeddings sit near a user's recent choices.

Ultralytics Platform uses vector search to grow training datasets. Its Find Similar Images action takes a selected image, searches public datasets and the user's own and team datasets for the nearest images by embedding, and shows up to 24 matches with their similarity scores, which can be added to the dataset as unlabeled training images.