Visual Question Answering (VQA)

Visual question answering (VQA) is the task of answering a natural-language question about an image, such as "What color is the bus?", "How many people are waiting?", or "Is the stove on?" The system receives the image and the question together and returns a short answer, so it has to read the question, find the relevant parts of the image, and combine them, which makes VQA a standard test of multimodal AI. Stanislaw Antol and colleagues at Virginia Tech and Microsoft Research introduced the task and its first large benchmark, the VQA dataset, at ICCV 2015: about 250,000 images, mostly from COCO, with 760,000 open-ended questions and 10 human answers to each, roughly 10 million answers in all.

How VQA Models Work#

A VQA model encodes the image with a convolutional neural network (CNN) or vision transformer (ViT), encodes the question with a language model, and fuses the two. Early models fused global image features with an LSTM's question vector and treated answering as classification over the few thousand most common answers. An attention mechanism then let the question choose where to look: Peter Anderson and colleagues' bottom-up and top-down model, which won the 2017 VQA Challenge, attended over regions proposed by an object detection model, so "What color is the bus?" could focus on the bus box. Current vision-language models (VLMs) generate free-form answers with a language model, after visual instruction tuning on image-question-answer data.

Benchmarks and the Language-Prior Problem#

Early VQA models scored well partly by ignoring the image. Many questions can be guessed from wording alone, as when "Is there a..." is usually answered "yes" and "What sport is..." is often "tennis." VQA v2, released by Yash Goyal and colleagues at CVPR 2017, countered these language priors by pairing each question with two similar images that have different answers, so a model has to look to score well. Later datasets each target a skill:

• VizWiz: Over 31,000 questions asked by blind people about photos they took with their phones, many of them blurry or poorly framed.
• TextVQA: Questions that require reading text in the scene, such as the number on a jersey or the name on a storefront.
• GQA: Compositional questions generated from scene graphs, such as "Is the cup to the left of the plate made of glass?"
• OK-VQA: Questions that need outside knowledge, such as which country a dish comes from.
• VQA-RAD: 3,515 clinician-written questions about 315 radiology images, built at the US National Library of Medicine.

Because people phrase correct answers differently, the VQA benchmark scores an answer by how many of the 10 human annotators gave it, counting three or more matches as fully correct and averaging over every subset of nine annotators:

# Ten crowd answers to "What color is the bus?", as collected for every VQA question
human_answers = ["red"] * 7 + ["red and white"] * 2 + ["orange"]


def vqa_accuracy(prediction, answers):
    """Average min(matches / 3, 1) over the ten leave-one-out subsets of nine answers."""
    scores = []
    for i in range(len(answers)):
        others = answers[:i] + answers[i + 1 :]
        scores.append(min(others.count(prediction) / 3, 1))
    return sum(scores) / len(scores)


for prediction in ["red", "red and white", "orange", "blue"]:
    print(f"{prediction}: {vqa_accuracy(prediction, human_answers):.2f}")

It prints 1.00 for "red", 0.60 for "red and white", 0.30 for "orange", and 0.00 for "blue": an answer only one annotator gave still earns partial credit, and an answer nobody gave earns none. Results are reported separately for yes/no, number, and other answer types, and number questions consistently score lowest, because counting needs every instance found exactly once. Detectors answer counting questions directly: YOLO26 returns one box per object, so the number of "person" boxes in its prediction result answers "How many people are in the picture?" for any of its 80 COCO classes. For classes outside that list, vision language models can return boxes too, and the VLM Leaderboard measures how accurately they detect objects in real-world datasets.

VQA vs. Related Tasks#

Image captioning describes a whole image without being asked anything, and VQA answers one targeted question. Visual grounding returns the location of the object a phrase names, often as a bounding box, and VQA returns an answer about that object. Optical character recognition (OCR) supplies the text that TextVQA-style questions ask about, and visual reasoning is the multi-step inference that compositional questions such as GQA's require. Text-only question answering works from a passage instead of pixels.

Applications#

Assistive technology is the clearest use. Be My Eyes added Be My AI, built on OpenAI's GPT-4, in 2023 so that blind and low-vision users can photograph a label, a screen, or a street and ask questions about it. Retail and manufacturing teams ask VLMs about shelf photos or inspection images instead of training a classifier for every question. In healthcare, medical VQA research targets questions such as "Is there a fracture?" about X-rays and CT scans, a form of medical image analysis that VQA-RAD measures with questions clinicians wrote while looking at the images.