Grounding

Grounding is the process of connecting words or other symbols to the things they refer to, so that an AI system's language is tied to evidence it can point at. In computer vision, visual grounding means locating the region of an image or video that a phrase describes: given "the man in the tan coat", a grounded model returns a bounding box or mask around that man. In large language models (LLMs), grounding means tying generated text to retrieved documents, search results, or tool outputs, so claims can be traced to a source. Both senses echo the symbol grounding problem that cognitive scientist Stevan Harnad posed in 1990: how symbols manipulated by a system can mean anything to that system rather than only to the people reading them.

Visual Grounding Tasks#

• Phrase grounding: Locate every object mentioned in a sentence, such as both "a woman" and "her dog" in a caption.
• Referring expression comprehension: Find the single object an expression singles out, such as "the second car from the left". The RefCOCO, RefCOCO+, and RefCOCOg benchmarks collect such expressions for COCO images, and RefCOCO+ bans location words like "left" so that expressions must describe appearance.
• Open-vocabulary detection: Find every instance of categories named in text at inference time, beyond the fixed class list of a closed-set object detection model.
• Grounded captioning and answers: A vision language model (VLM) writes a description or answer and attaches a box to each object it mentions.
• Temporal and 3D grounding: Find the moment in a video that a sentence describes, or the object in a 3D scan of a room.

How Grounding Models Work#

A grounding model encodes the text and the image, scores every candidate image region against every phrase, and keeps the regions whose scores pass a threshold. Training aligns the two embedding spaces, often with contrastive learning on image-text pairs, so that the embedding of "blue backpack" lands near the features of blue backpacks. Stronger models also fuse the modalities early with cross-attention, letting the words steer which visual features the network computes.

GLIP recast object detection itself as phrase grounding, so detection and grounding data could train one model, and Grounding DINO, which Ultralytics Platform hosts for auto-annotation, accepts category names or referring expressions and reached 52.5 AP on COCO without any COCO training data. Meta's SAM 3, released in November 2025 and hosted on Platform as SAM 3 Text, segments and tracks every instance of a short noun phrase such as "yellow school bus" across images and video. Generative VLMs ground by writing box coordinates as text or as special location tokens alongside their answers. The VLM Leaderboard compares how accurately open-vocabulary detectors and generative VLMs ground the class names of real-world detection datasets.

YOLOE, introduced by Tsinghua University researchers and available in the Ultralytics package as YOLOE-26 on the YOLO26 architecture, grounds category names and short phrases in real time: a text encoder turns each phrase into an embedding, and the detection head matches it against every image region:

from ultralytics import YOLOE

# Load an open-vocabulary YOLOE-26 detection and segmentation model
model = YOLOE("yoloe-26s-seg.pt")

# Each phrase is embedded once by the text encoder and becomes a class the model can find
model.set_classes(["bus", "person", "sunglasses", "traffic sign"])
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Print each grounded phrase with its pixel box (x1, y1, x2, y2)
for cls, box in zip(result.boxes.cls.tolist(), result.boxes.xyxy.tolist()):
    print(result.names[int(cls)], [round(v) for v in box])

Each printed line pairs one of the phrases with the region YOLOE grounded it to, and a phrase with no match in the image prints nothing.

Applications#

• Robotics: A robot told to "pick up the red mug next to the laptop" must ground that phrase to one object and its 3D position before planning a grasp.
• Data annotation: Open-vocabulary detectors draft boxes and masks for new classes from a list of names, which annotators then correct instead of drawing from scratch. Ultralytics Platform auto-annotates detection datasets this way before a YOLO26 model trains on the corrected labels.
• Video search: Temporal grounding retrieves the clip where "a cyclist turns left at the intersection" from hours of footage.
• Accessibility: Assistive apps answer "where is the exit sign?" by locating the object, not only naming it.

Grounding vs. Related Terms#

• Image captioning: Maps an image to text. Grounding runs the other way, from text to the image regions it describes.
• Visual question answering (VQA): Answers a question about an image in words. Grounded VQA also points to the evidence for the answer.
• Zero-shot learning: Recognizes classes with no training examples. Open-vocabulary grounding is one way to achieve it, since a class described in words needs no labeled images.
• Multimodal AI: The broader field of models that combine data types; grounding is the alignment of specific words with specific parts of the other modality.

Grounding in Language Models#

An LLM that answers from memory alone can produce fluent claims with no source, one cause of hallucination. Grounded generation supplies evidence at inference time, through retrieval-augmented generation (RAG), web search, database queries, or tool calls, and asks the model to answer from that evidence and cite it. A grounded answer can be checked claim by claim against its sources, and evaluations of grounded systems measure faithfulness, whether each statement is supported by the retrieved text, separately from whether the retrieved text was correct.