License YOLO
Back to Ultralytics Glossary

ColBERT

ColBERT encodes queries and documents as per-token BERT embeddings and scores them by late interaction (MaxSim) for fast, accurate search and RAG.

ColBERT (Contextualized Late Interaction over BERT) is a neural retrieval model that represents a query and each document as one embedding per token and scores a document by late interaction: every query token finds its most similar document token, and those maximum similarities, called MaxSim, are summed into the relevance score. Omar Khattab and Matei Zaharia of Stanford University introduced it in the SIGIR 2020 paper ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Document token embeddings are computed once, offline, so the work at query time is a cheap similarity computation instead of a full BERT pass for every candidate document.

Where Late Interaction Fits#

Neural retrievers differ in when the query and document meet. Bi-encoders compress each text into a single vector, the standard setup for vector search: document vectors are precomputed and search is fast, but one vector blurs the distinctions between individual terms. Cross-encoders feed the query and document through a transformer together, so attention compares every pair of tokens. They are the most accurate and must run once per query-document pair, which limits them to reranking short candidate lists.

ColBERT encodes the query and the document separately, as a bi-encoder does, but keeps one 128-dimensional vector per token and lets the two sides interact only in the final MaxSim step. Queries are padded with BERT's mask token to a fixed length of 32, which gives the model extra slots to expand the query with related terms. On the MS MARCO and TREC CAR passage-search benchmarks, ColBERT was competitive with BERT-based rankers while running two orders of magnitude faster and using four orders of magnitude fewer FLOPs per query.

Computing a MaxSim Score#

The MaxSim operator is a few lines of PyTorch. This example applies it to contextual token embeddings from a small BERT-family encoder:

import torch.nn.functional as F
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
query = "who wrote the novel Dune"
docs = [
    "Frank Herbert published the science fiction novel Dune in 1965.",
    "A dune is a hill of sand shaped by the wind.",
]

# One L2-normalized contextual embedding per query token
q = F.normalize(encoder.encode(query, output_value="token_embeddings"), dim=-1)
for doc in docs:
    d = F.normalize(encoder.encode(doc, output_value="token_embeddings"), dim=-1)

    # MaxSim: each query token keeps its best-matching document token, and the maxima are summed
    score = (q @ d.T).max(dim=1).values.sum()
    print(f"{score.item():.2f}  {doc}")

The novel passage scores 5.90 and the sand-dune passage 4.33. Both contain "dune," which matches strongly in each, but in the first passage the query token "wrote" finds "published" with a cosine similarity of 0.72 and "novel" finds itself, while in the second passage the best match for "wrote" reaches only 0.29.

  • BERT itself: The pretrained encoder that ColBERT fine-tunes for retrieval, adding a linear layer that projects each token to 128 dimensions.
  • Late chunking: Also encodes text in context before splitting it, then averages the tokens into one vector per chunk, while ColBERT keeps every token vector.
  • Semantic search: The task ColBERT performs, ranking passages by meaning rather than by shared keywords.
  • Hybrid search: Merges keyword and single-vector results to get exact-term matching and meaning together. ColBERT's token-level matching captures part of that exactness inside one neural model.

Late Interaction for Images#

Late interaction also works on images. ColPali (2024) embeds each document page screenshot as a grid of patch vectors with a vision-language model and scores them against query tokens with MaxSim, bringing late interaction to multimodal RAG over PDFs. Open-vocabulary detection keeps image features fine-grained for a related reason: Ultralytics YOLOE stores a text embedding for each prompted class, such as "double-decker bus," and compares it with the region features at every location of its feature maps, so a short prompt can match one small object instead of a single vector that summarizes the whole image.

ColBERTv2 and PLAID#

Storing a vector for every token makes a ColBERT index roughly an order of magnitude larger than a single-vector index, a cost that matters when ColBERT serves as the first-stage retriever or reranker in a retrieval-augmented generation (RAG) pipeline. The Stanford ColBERT repository implements two answers. PLAID is a search engine that scores candidates against cluster centroids before running full MaxSim. ColBERTv2 (NAACL 2022) stores each token vector as the ID of its nearest centroid plus a residual quantized to 1 or 2 bits per dimension and trains with denoised supervision distilled from a cross-encoder, and its residual compression cut the space footprint of late interaction by 6 to 10 times.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning