Multimodal RAG

Multimodal RAG is retrieval-augmented generation (RAG) over a knowledge base that holds images, video, audio, and scanned or rendered documents as well as text. A question retrieves the most relevant items from any of these modalities, and a vision-language model (VLM) generates the answer from them. A wiring diagram, a product photo, or a chart inside a PDF can then be retrieved and read directly, instead of only through a text description someone wrote beforehand.

How Non-Text Data Gets Indexed#

The simplest approach converts everything to text. Image captioning, optical character recognition (OCR), speech-to-text transcripts, and the labels from object detection describe each item in words that an ordinary text index can search. This reuses a text RAG pipeline unchanged and loses whatever the description leaves out.

The second approach embeds images and text in one shared space. Models trained with contrastive learning, such as OpenAI's CLIP and Google's SigLIP, map a photo and a sentence describing it to nearby embeddings, so a text question retrieves images directly from a vector database. For documents, ColPali (2024) embeds each page screenshot as a grid of patch vectors and scores it against the question's tokens with ColBERT-style late interaction, which skips OCR and layout parsing for PDFs full of tables and charts.

Generation then passes the retrieved items, often a mix of images and text passages, to the VLM together with the question. Grounding the answer in retrieved images reduces hallucinations about visual details the model never saw during training, such as the layout of one company's control panel.

Retrieval in a Shared Embedding Space#

This example uses the CLIP ViT-B/32 checkpoint to pick the image that answers each question from a two-image knowledge base:

import requests
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Image knowledge base: CLIP embeds images and text into one shared vector space
urls = ["https://ultralytics.com/images/bus.jpg", "https://ultralytics.com/images/zidane.jpg"]
images = [Image.open(requests.get(url, stream=True, timeout=10).raw) for url in urls]
questions = ["What color is the man's tie?", "How many people are standing next to the bus?"]

# Retrieve: score each question against every image and keep the best match for the VLM
inputs = processor(text=questions, images=images, return_tensors="pt", padding=True)
scores = model(**inputs).logits_per_text
for question, row in zip(questions, scores):
    print(f"{question} -> {urls[row.argmax()].split('/')[-1]}")

The tie question retrieves zidane.jpg and the bus question retrieves bus.jpg, because CLIP places each question's text embedding closest to the photo it is about. The Ultralytics LLM interface covers the generation step that follows: llm(question, image=...) sends the question and the retrieved image to any OpenAI-compatible vision-language model in one call and returns the provider's response.

Multimodal RAG vs. Related Terms#

• Visual question answering (VQA): Answers a question about an image the user supplies. Multimodal RAG first finds which images, pages, or clips to look at among thousands.
• Multimodal models: The generators, such as OpenAI's GPT or Google's Gemini models, that read images and text together. Multimodal RAG is the retrieval system that decides what those models see.
• Semantic search: The retrieval half on its own, returning ranked results to a person instead of to a model.

Real-World Applications#

• Manufacturing maintenance: A technician photographs a failed part, and the system retrieves the manual page with the matching exploded diagram plus photos from past repairs of the same assembly.
• Retail search and support: A shopper's photo retrieves visually similar catalog items, and the assistant answers fit or compatibility questions from their product sheets.
• Document question answering: Annual reports, scientific papers, and slide decks keep key numbers in charts and tables, which page-image retrieval finds without a separate table parser.
• Video understanding: Indexing sampled frames or short clips by embedding lets a question such as "When did the delivery van arrive?" retrieve the right minutes of footage, so the VLM reads a handful of frames instead of hours of video.

Design Trade-Offs#

Images are expensive context: a VLM turns each image into hundreds or thousands of tokens, so pipelines usually pass only the top few retrieved images and surround them with shorter text passages. Shared-embedding retrievers also inherit limits from their training. CLIP's text encoder reads at most 77 tokens, so longer questions are truncated before matching, and whole-image embeddings capture overall scene content better than small details such as a serial number on a label. Text conversion keeps those details searchable whenever OCR can read them, so many systems index both OCR text and image embeddings and merge the two result lists with hybrid search.