License YOLO
Back to Ultralytics Glossary

Vision Language Model (VLM)

A vision language model (VLM) processes images and text together to caption, answer questions, and find objects by name. Architectures and examples.

A vision language model (VLM) is a neural network that processes images, and often video, together with text. Depending on its design, it scores how well an image matches a caption, writes text about an image, or finds the objects a text prompt names. VLMs answer questions such as "How many people are waiting at the bus stop?", turn charts and receipts into structured data, and locate objects described in free-form words. It joins the visual perception of computer vision models, which predict from a fixed list of classes, with the open-ended language of large language models (LLMs).

How Vision Language Models Work#

Every VLM needs a vision encoder, usually a Vision Transformer (ViT), that turns an image into embeddings, and a way to relate those embeddings to text. Three designs are common:

  • Dual encoders: CLIP and SigLIP encode images and text separately into one shared embedding space, trained with contrastive learning on hundreds of millions to billions of image-caption pairs. They measure similarity rather than generate text, which makes them the standard tools for zero-shot classification and image search.
  • Generative VLMs: A pretrained vision encoder, often from CLIP or SigLIP, feeds a projector that converts its patch embeddings into "visual tokens" in the LLM's input space. The LLM reads them alongside the text prompt and writes the answer. LLaVA-1.5 works this way: its CLIP ViT-L/14 encoder splits a 336×336 image into a 24×24 grid of patches, producing 576 visual tokens. DeepMind's Flamingo took a different route, inserting cross-attention layers into a frozen LLM so text tokens attend to image features.
  • Open-vocabulary detectors: Models such as Grounding DINO, YOLO-World, and YOLOE compare image regions with text embeddings to return boxes or masks for any named object, a task called grounding. Some generative VLMs ground objects too, writing box coordinates as text or as special location tokens.

In the generative design, the LLM writes its answer one token at a time, attending to the visual tokens and the prompt at every step:

Generative VLMs usually train in stages. The projector first learns to align visual tokens with the LLM's embeddings on image-caption pairs while the encoder and LLM stay frozen. Visual instruction tuning then fine-tunes the model on conversations about images, and preference tuning such as RLHF reduces answers that describe things the image does not contain.

Open-vocabulary detectors bring the same text matching to real-time detection. YOLOE, introduced by Tsinghua University researchers and available in the Ultralytics package as YOLOE-26 on the YOLO26 architecture, turns class names into embeddings with Apple's MobileCLIP2 text encoder and matches them against image regions, returning boxes and instance masks for categories it was never trained to label. The prompts below include sunglasses, which is not one of the 80 COCO classes that closed-set YOLO models detect:

from ultralytics import YOLOE

# Load an open-vocabulary YOLOE-26 detection and segmentation model
model = YOLOE("yoloe-26s-seg.pt")

# Name the classes in plain words; the text encoder turns each into an embedding
model.set_classes(["person", "glasses", "bus"])

# Detect and segment only the prompted classes
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]
print([result.names[int(c)] for c in result.boxes.cls])
result.save(filename="yoloe_prompts.jpg")

The printed list contains the bus, four people, and the glasses worn by two of them, and changing the list of words changes what the model detects without any retraining.

Real-World Applications#

  • Visual question answering: Inspection, retail, and support tools ask a VLM about a photo, such as whether a shelf is fully stocked or which warning light is on in a dashboard image.
  • Document understanding: VLMs read invoices, forms, tables, and charts directly from page images and return structured fields, often replacing a separate OCR step.
  • Image captioning and accessibility: Generated descriptions supply alt text for screen readers and searchable metadata for photo libraries.
  • Robotics: Vision-language-action models such as Google DeepMind's RT-2 extend a VLM to output robot actions, so a robot can follow instructions like "pick up the empty can".
  • Data labeling: Open-vocabulary detectors draft boxes and masks for new classes from text prompts, giving annotators a starting point instead of a blank image. Ultralytics Platform auto-annotates detection datasets with such detectors and with generative VLMs before a YOLO26 model trains on the labels.
  • LLM: A text-only model. Many generative VLMs are an LLM with a vision encoder attached, and they inherit its reasoning, its knowledge, and its tendency to make things up.
  • Multimodal AI: The broader category of models that combine data types. A model that pairs audio with text, or radar with camera images, is multimodal without being a VLM.
  • Object detection: A closed-set detector predicts only the classes it was trained on and has no text input. Open-vocabulary detectors are the VLMs that take over this task when the classes are named at inference time. The VLM Leaderboard compares how accurately open-vocabulary detectors and generative VLMs detect objects in real-world datasets.

Limitations#

VLMs hallucinate: a captioner can mention a fork on a table that holds none, because the language model completes a likely sentence. Fine spatial reasoning, counting large groups, and reading small text are further weak points, and every image costs hundreds or thousands of tokens of the LLM's context window. The POPE benchmark measures object hallucination by polling a model with questions of the form "Is there a car in the image?", drawing the absent objects at random, from the most frequent objects in the dataset, or from objects that often appear alongside the ones present.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning