VLM Leaderboard

Sample. Suite ul37-v2 takes 10 labeled images at random from each of the 36 UL37 detection datasets on Ultralytics Platform, plus 10 COCO 2017 images as a reference: 370 images and 7,268 boxes. The draw is reproducible: each dataset contributes the images with the lowest hash of the suite version and image. Images are used whole, with no crops or limits on box count and size, so crowded and tiny-object images appear as often as in the datasets; images with ignore regions are left out. MODA, the only oriented-box dataset, is excluded.

Class names. Each dataset is sent with its own class names on Ultralytics Platform, which are readable, for example DDR's EX is "hard exudate", and RPC's 200 products each carry a short description.

Inference. Each image goes through production annotation with its dataset's class names and name, exactly as Smart and Batch Annotation call it, and SAM 3.1 tightens every model's boxes except SAM 3's, as it does for users. A failed request (timeout, refusal or invalid output) counts as an image with no predictions; when output is cut off, the boxes returned before the cut are scored.

Scoring. Every box is scored with the Ultralytics package's own detection metrics, the code that validates YOLO models. Models return no confidence scores, so each box ranks by where the model listed it, earlier meaning more confident. mAP₅₀ averages AP over the labeled classes at IoU 0.5, mAP_50-95 averages IoU 0.50 to 0.95, and precision and recall count every box at IoU 0.5. Each dataset pools its 10 images and datasets are averaged equally. The COCO 2017 images are only a sanity check: they are scored apart from the average and show in each model's per-dataset scores.

Cost and speed. Cost is the provider-reported token usage priced at Ultralytics Platform rates, per 1,000 successfully annotated images. Speed is the median request time per image. Every provider runs at its default reasoning effort.

Limits. 10 images per dataset are enough to rank models and domains, not to certify a single dataset's score. Models sample stochastically, so repeated runs move individual numbers by a few points.