MMMU

MMMU stands for Massive Multi-discipline Multimodal Understanding and Reasoning, a benchmark that tests how well AI models solve college-level problems combining images, text, and subject knowledge. Think of it as an exam in which understanding a diagram, chart, or chemical structure is essential to answering the question. It evaluates vision-language models, which process visual and written information together.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, and colleagues introduced the MMMU benchmark on November 27, 2023. Its central challenge is combining visual perception with specialized knowledge and deliberate reasoning.

What MMMU Measures#

MMMU brings together three abilities: perceiving visual information, recalling relevant knowledge, and reasoning toward an answer. This makes it a test of multimodal AI, where different information types contribute to one task. Its visual reasoning problems extend across diagrams, tables, maps, sheet music, medical images, and technical blueprints. Understanding these representations requires subject-specific interpretation as well as recognizing their visible contents.

MMMU is a specific benchmark dataset within visual question answering, the broader task of answering questions about images. MMLU evaluates text-based academic knowledge; MMMU incorporates visual evidence into academic problem-solving. Optical character recognition extracts written characters from images, while MMMU questions also require interpreting relationships and applying domain knowledge.

Dataset Size and Evaluation Structure#

The MMMU dataset contains 11,550 questions, commonly rounded to 11.5K. Questions span 30 subjects and 183 subfields across six disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. College students manually collected the material from sources including exams, quizzes, textbooks, and lecture materials.

Its evaluation structure assigns a distinct purpose to each split:

• Few-shot learning: The 150-question development set supplies examples that can be included in a model's prompt.
• Validation data: The 900-question validation set supports debugging, parameter selection, and quick evaluations.
• Test data: The 10,500-question test set supports broader assessment. Released test answers and explanations enable local scoring with the official evaluation implementation.

Scores, Applications, and Failure Modes#

MMMU reports accuracy: the percentage of questions answered correctly. The original headline result was approximately 56% for GPT-4V; its detailed test score was 55.7%, compared with 56.8% on validation. These initial evaluations used zero-shot learning, presenting questions without benchmark demonstrations. Split and prompting configuration therefore identify the evaluation behind a leaderboard score.

A useful application is diagnosing why multimodal answers fail. In an analysis of 150 GPT-4V errors, 35% involved perception, 29% involved missing knowledge, and 26% involved reasoning. One computer science example showed the model recognizing double circles in a finite-state diagram but failing to interpret them as accepting states. The visible symbol was recognized; its technical meaning was lost, producing an incorrect solution. This connects MMMU to grounding, the association between model interpretations and supporting evidence.

The same image-and-language capabilities support class-prompted data annotation in Ultralytics Platform, where models generate bounding boxes from class names for detection datasets. Its annotation editor lets people review and refine those predictions.

MMMU vs. MMMU-Pro#

Xiang Yue, Tianyu Zheng, Yuansheng Ni, and colleagues introduced MMMU-Pro in September 2024 as a harder version of MMMU. It targets shortcut learning: exploiting textual clues or answer-option correlations to solve questions without using the intended visual evidence. The construction process filters questions answerable by text-only models, expands answer choices, and adds a vision-only input setting.

In MMMU-Pro, vision-only input places the question and answer choices inside a screenshot or photograph, requiring the model to read and interpret a combined visual presentation. This connects document understanding with academic reasoning. The original construction retained 1,730 questions, with candidate answers expanded to 10 options.