VBench
VBench scores AI-generated video on 16 dimensions, from subject consistency to prompt faithfulness. What each one measures and how to read the results.
VBench is a benchmark suite for text-to-video generation that scores generated clips on 16 separate dimensions of visual quality and prompt faithfulness. Researchers from Nanyang Technological University's S-Lab and Shanghai AI Laboratory introduced it in November 2023 in the paper VBench: Comprehensive Benchmark Suite for Video Generative Models, which was selected as a highlight paper at CVPR 2024. Its public leaderboard has since become a common reference point for comparing video generation models, from the LaVie, ModelScope, VideoCrafter, and CogVideo models in the original paper to later diffusion transformer systems.
Before VBench, evaluation leaned on single distribution metrics such as Fréchet Video Distance, the video counterpart of Fréchet Inception Distance (FID). Those metrics compare feature statistics of real and generated clips, so one number has to stand in for motion, identity, and prompt following at once. VBench breaks quality into parts so developers can see exactly where a model falls short, and its authors collected human preference annotations for every dimension, comparing pairs of videos from different models, to confirm that each automatic score ranks models the way viewers do.
The 16 Dimensions#
The VBench evaluation dimensions fall into two groups:
- Video quality judges the clip on its own terms, without the prompt: subject consistency, background consistency, temporal flickering, motion smoothness, and dynamic degree for quality over time, plus aesthetic quality and imaging quality for each frame.
- Video-condition consistency checks the clip against its prompt: object class, multiple objects, human action, color, spatial relationship, and scene for content; appearance style and temporal style for style; and overall consistency.
Dynamic degree balances the consistency metrics. A nearly static clip scores well on subject consistency and temporal flickering almost by default, so VBench also measures how much motion actually occurs. Reading these dimensions together stops a model from winning by generating moving photographs.
How Each Dimension Is Scored#
Each dimension has its own evaluator, usually a pretrained vision model applied to sampled frames. Subject consistency compares features from DINO, a self-supervised vision transformer, across frames, and background consistency does the same with CLIP image features. Temporal flickering is the mean absolute pixel difference between consecutive frames, measured only on near-static clips selected with RAFT optical flow, which also estimates motion for dynamic degree. Motion smoothness checks how well a frame interpolation model, AMT, reconstructs dropped frames, since natural motion is predictable from its neighbors. Aesthetic quality uses the LAION aesthetic predictor, and imaging quality uses the MUSIQ model to detect blur, noise, and overexposure.
The content dimensions run other models over the frames. Object class, multiple objects, color, and spatial relationship use the GRiT object detection and dense captioning model to confirm that the requested objects appear in the right colors and arrangement. Human action uses the UMT action recognition model, scene compares the prompt with image captions from Tag2Text, appearance style compares frames with the style description using CLIP, and temporal style and overall consistency compare video and text embeddings from ViCLIP.
Each dimension draws on a prompt suite of about 100 prompts written to test it. A separate set of 800 prompts, 100 in each of eight categories (animal, architecture, food, human, lifestyle, plant, scenery, and vehicles), lets results be broken down by subject matter. The leaderboard normalizes each dimension and combines them into a weighted quality score, a weighted semantic score, and a total score.
Using VBench Results#
Leaderboard scores are comparable only when models are run on the standard prompt suite with the same sampling settings. VBench can also score custom videos outside that suite, but only on the six dimensions that need no reference prompt: subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality.
Because every dimension is measured by another model, each score inherits that model's blind spots. A detector that misses small or unusual objects undercounts them in generated video too, and an aesthetic predictor reflects the taste of the data it was trained on. As generators improved on the original appearance-focused dimensions, the same group extended the suite. VBench++, published in IEEE TPAMI, adds image-to-video, long-video, and trustworthiness evaluation covering fairness, bias, and safety, and VBench-2.0, released in March 2025, scores 18 dimensions of "intrinsic faithfulness" grouped into human fidelity, controllability, creativity, physics, and commonsense.
The detector-based checks carry over when generated clips become synthetic training data for a vision model. Running Ultralytics YOLO26 in predict mode over each generated frame confirms that the requested objects actually appear before the footage joins a training set, the same check VBench's object class dimension makes with GRiT.










