ColBERT
ColBERT encodes queries and documents as per-token BERT embeddings and scores them by late interaction (MaxSim) for fast, accurate search and RAG.
ColBERT (Contextualized Late Interaction over BERT) is a neural retrieval model that represents a query and each document as one embedding per token and scores a document by late interaction: every query token finds its most similar document token, and those maximum similarities, called MaxSim, are summed into the relevance score. Omar Khattab and Matei Zaharia of Stanford University introduced it in the SIGIR 2020 paper ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Document token embeddings are computed once, offline, so the work at query time is a cheap similarity computation instead of a full BERT pass for every candidate document.
Where Late Interaction Fits#
Neural retrievers differ in when the query and document meet. Bi-encoders compress each text into a single vector, the standard setup for vector search: document vectors are precomputed and search is fast, but one vector blurs the distinctions between individual terms. Cross-encoders feed the query and document through a transformer together, so attention compares every pair of tokens. They are the most accurate and must run once per query-document pair, which limits them to reranking short candidate lists.
ColBERT encodes the query and the document separately, as a bi-encoder does, but keeps one 128-dimensional vector per token and lets the two sides interact only in the final MaxSim step. Queries are padded with BERT's mask token to a fixed length of 32, which gives the model extra slots to expand the query with related terms. On the MS MARCO and TREC CAR passage-search benchmarks, ColBERT was competitive with BERT-based rankers while running two orders of magnitude faster and using four orders of magnitude fewer FLOPs per query.
Computing a MaxSim Score#
The MaxSim operator is a few lines of PyTorch. This example applies it to contextual token embeddings from a small BERT-family encoder:
import torch.nn.functional as F
from sentence_transformers import SentenceTransformer
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
query = "who wrote the novel Dune"
docs = [
"Frank Herbert published the science fiction novel Dune in 1965.",
"A dune is a hill of sand shaped by the wind.",
]
# One L2-normalized contextual embedding per query token
q = F.normalize(encoder.encode(query, output_value="token_embeddings"), dim=-1)
for doc in docs:
d = F.normalize(encoder.encode(doc, output_value="token_embeddings"), dim=-1)
# MaxSim: each query token keeps its best-matching document token, and the maxima are summed
score = (q @ d.T).max(dim=1).values.sum()
print(f"{score.item():.2f} {doc}")The novel passage scores 5.90 and the sand-dune passage 4.33. Both contain "dune," which matches strongly in each, but in the first passage the query token "wrote" finds "published" with a cosine similarity of 0.72 and "novel" finds itself, while in the second passage the best match for "wrote" reaches only 0.29.
ColBERT vs. Related Terms#
- BERT itself: The pretrained encoder that ColBERT fine-tunes for retrieval, adding a linear layer that projects each token to 128 dimensions.
- Late chunking: Also encodes text in context before splitting it, then averages the tokens into one vector per chunk, while ColBERT keeps every token vector.
- Semantic search: The task ColBERT performs, ranking passages by meaning rather than by shared keywords.
- Hybrid search: Merges keyword and single-vector results to get exact-term matching and meaning together. ColBERT's token-level matching captures part of that exactness inside one neural model.
Late Interaction for Images#
Late interaction also works on images. ColPali (2024) embeds each document page screenshot as a grid of patch vectors with a vision-language model and scores them against query tokens with MaxSim, bringing late interaction to multimodal RAG over PDFs. Open-vocabulary detection keeps image features fine-grained for a related reason: Ultralytics YOLOE stores a text embedding for each prompted class, such as "double-decker bus," and compares it with the region features at every location of its feature maps, so a short prompt can match one small object instead of a single vector that summarizes the whole image.
ColBERTv2 and PLAID#
Storing a vector for every token makes a ColBERT index roughly an order of magnitude larger than a single-vector index, a cost that matters when ColBERT serves as the first-stage retriever or reranker in a retrieval-augmented generation (RAG) pipeline. The Stanford ColBERT repository implements two answers. PLAID is a search engine that scores candidates against cluster centroids before running full MaxSim. ColBERTv2 (NAACL 2022) stores each token vector as the ID of its nearest centroid plus a residual quantized to 1 or 2 bits per dimension and trains with denoised supervision distilled from a cross-encoder, and its residual compression cut the space footprint of late interaction by 6 to 10 times.










