Visual Reasoning

Visual reasoning is an AI model's ability to reach a conclusion that requires several inferential steps over visual information: counting objects that meet a condition, comparing their sizes or positions, following spatial relations, applying physical common sense, or completing an abstract visual pattern. Recognizing a mug takes one step of perception, and answering "Is the mug to the left of the largest object made of the same material as the table?" takes several, because the answer combines facts about four objects. The capability sits between computer vision, which supplies those facts, and the reasoning models that chain inferences in text, and it is what visual question answering (VQA) benchmarks probe when their questions go beyond naming what is in the picture.

Benchmarks That Define Visual Reasoning#

Visual reasoning tests predate AI. John C. Raven's Progressive Matrices (1938) ask people to pick the tile that completes a 3×3 pattern, and Mikhail Bongard's 1967 problems ask which rule separates six images from six others. Machine benchmarks borrowed both ideas and added language:

• CLEVR: Justin Johnson and colleagues at Stanford and Facebook AI Research released this diagnostic dataset at CVPR 2017, with 100,000 rendered scenes of simple 3D shapes and about one million generated questions that test counting, comparison, spatial relations, and logic while minimizing the dataset biases that let models guess.
• RAVEN: A 2019 dataset of procedurally generated Raven-style matrices for training and testing machine pattern completion.
• ARC: François Chollet's 2019 Abstraction and Reasoning Corpus, grid puzzles whose transformation rule must be inferred from a few examples.
• MathVista: A 2023 benchmark of math problems posed in charts, plots, geometry diagrams, and photos.
• BLINK: A 2024 set of 14 classic vision problems, including relative depth and visual correspondence, recast as multiple-choice questions.

BLINK showed how far fluent multimodal models still were from human perception: people averaged 95.70 percent, while GPT-4V scored 51.26 percent and Gemini 45.72 percent, only 13.17 and 7.63 points above random guessing, and specialist vision models solved the same problems much better.

Approaches to Visual Reasoning#

End-to-end vision-language models (VLMs) answer directly from image tokens, and chain-of-thought prompting makes them write out intermediate steps before answering. OpenAI's o3 and o4-mini, released in April 2025, extended this to thinking with images: during reasoning, the model can crop, zoom, and rotate the image and look again at the result.

Neuro-symbolic methods separate seeing from reasoning. Jacob Andreas and colleagues' neural module networks (CVPR 2016) assembled a small network for each question from reusable modules such as "attend" and "re-attend." Kexin Yi and colleagues' NS-VQA (NeurIPS 2018) parsed each CLEVR scene into a table of objects with their attributes and positions, translated the question into a program, and ran the program on the table, reaching 99.8 percent accuracy with a fully traceable answer, an approach rooted in symbolic AI. Visual programming brought the same idea to real photos: VisProg, which won the CVPR 2023 best paper award, and ViperGPT (ICCV 2023) have a large language model (LLM) write a Python program that calls an object detection model, a depth estimation model, and other vision tools, then executes it.

Detection and Depth as Reasoning Inputs#

Relative depth is one of the BLINK tasks, and it is exactly the kind of question a visual program delegates to a specialist model. YOLO26 supplies both inputs: an object detection model turns the photo into named boxes, and a monocular depth model returns a per-pixel depth map in meters, so a few lines of logic answer "Is anyone closer to the camera than the bus?":

from ultralytics import YOLO

image = "https://ultralytics.com/images/bus.jpg"
detections = YOLO("yolo26n.pt")(image)[0]
depth = YOLO("yolo26n-depth.pt")(image)[0].depth.data  # (H, W) depth map in meters

# Perception to symbols: one (name, median depth) fact per detected object
boxes = detections.boxes.xyxy.int().tolist()
classes = detections.boxes.cls.int().tolist()
facts = []
for (x1, y1, x2, y2), cls in zip(boxes, classes):
    facts.append((detections.names[cls], depth[y1:y2, x1:x2].median().item()))

# Reasoning over symbols: compare every person's depth with the bus
bus_depth = min(d for name, d in facts if name == "bus")
closer = [round(d, 1) for name, d in facts if name == "person" and d < bus_depth]
print(f"Bus at {bus_depth:.1f} m; people closer to the camera: {closer}")

Each answer can be traced back to a specific box and depth value, which an end-to-end model's answer cannot. The same pattern scales to robotics and spatial intelligence tasks, where a robot must decide which object to reach for first, and to agentic workflows that check a scene before acting.

Visual Reasoning vs. Related Terms#

VQA is a task format, a question about an image with a short answer, and visual reasoning is the multi-step capability that harder VQA questions require. Multimodal AI covers any model that combines data types, whether or not it reasons. Spatial reasoning is the subset concerned with positions, distances, and orientation, and its simplest questions need no language model at all: BLINK's relative depth items ask which of two marked points is closer to the camera, which a depth map answers by comparing two pixel values.