Zero-Shot Learning

Zero-shot learning (ZSL) is the ability of a model to recognize classes for which it has seen no labeled training examples. Instead of examples, the model receives a description of each new class, such as a list of attributes or, in modern systems, the class name as text, and it matches inputs against those descriptions in a shared embedding space. Standard supervised learning can only predict the classes in its training labels, so adding a category means collecting and labeling new data and retraining. A zero-shot model adds a category by adding its name. OpenAI's CLIP, trained on 400 million image-caption pairs, matched the ImageNet accuracy of the original supervised ResNet-50 without using any of ImageNet's 1.28 million labeled training images.

How Zero-Shot Learning Works#

Zero-shot learning transfers knowledge through a semantic space that links seen and unseen classes. Early methods described every class with human-defined attributes, such as "striped," "has hooves," and "lives in grasslands," and trained models to predict those attributes, so an animal never seen in training could be recognized as the class whose attributes it matched. Vision-language models replaced hand-built attributes with natural language. CLIP trains an image encoder and a text encoder with contrastive learning so that matching images and captions get similar embeddings. To classify, it embeds a prompt for each candidate class, such as "a photo of a zebra," embeds the image, and picks the class with the highest cosine similarity:

Because the classifier is built from text at inference time, the list of classes can change with every request. The same pattern carries over to large language models (LLMs), where zero-shot prompting means giving the model only an instruction, with no worked examples, as defined in the GPT-3 paper.

Zero-Shot Object Detection#

Open-vocabulary detectors extend zero-shot learning from whole images to individual objects. They score every candidate region against text embeddings of the requested classes, so they can find objects outside any fixed label set, such as a specific product in a video archive. YOLO-World applies this approach to real-time YOLO models. Ultralytics YOLOE-26, an open-vocabulary version of YOLO26, encodes each class name passed to set_classes() with a MobileCLIP2 text encoder and returns boxes and instance segmentation masks for those classes without retraining:

from ultralytics import YOLOE

# YOLOE-26 checkpoint that accepts text prompts
model = YOLOE("yoloe-26s-seg.pt")

# Name the classes at inference time, here with words that are not COCO labels
model.set_classes(["city bus", "sneakers", "glasses"])

# Detect and segment the named classes
results = model.predict("https://ultralytics.com/images/bus.jpg")
result = results[0]
result.save(filename="zero_shot_result.jpg")

The first set_classes() call downloads the text encoder. For object detection, "zero-shot" means the model received no labeled boxes for a class, although the class name may have appeared in the image-text data used to pretrain its encoders. Ultralytics Platform auto-annotates detection datasets with zero-shot detectors and vision language models, and the VLM Leaderboard compares how accurately each one finds objects in real-world images.

Applications#

• Visual search and moderation: Media libraries and content platforms find images that match a text query, or flag new categories of content, without training a classifier for each query.
• Robotics: A robot told to fetch "the blue mug" can locate an object category it was never trained to detect.
• Wildlife and biodiversity: BioCLIP, trained on more than 10 million images across the tree of life, classifies species zero-shot from their names, and won a best student paper award at CVPR 2024.
• Medical imaging: Models trained on X-rays paired with their written radiology reports can flag findings that were never annotated as explicit training labels.

Challenges#

Zero-shot accuracy is usually lower than that of a model trained on labeled examples of the same classes, especially for fine-grained or specialized categories whose names are rare in web text. In generalized zero-shot learning, where test images can come from both seen and unseen classes, models favor the classes they were trained on, and benchmark studies found that accuracy on unseen classes falls sharply in that setting. Results also depend on wording, which makes prompt engineering part of zero-shot classification: CLIP's ImageNet accuracy rose by 1.3% when the bare class name was replaced with the prompt "a photo of a {label}."

Zero-Shot Learning vs. Related Terms#

• Few-shot learning: Learns new classes from a handful of labeled examples per class, typically 1 to 10. Zero-shot learning is the case with zero examples and a description instead.
• One-shot learning: The few-shot case with exactly one labeled example per class, often matched by embedding similarity.
• Transfer learning: Reuses a pretrained model for a new task, usually by fine-tuning on labeled data. Zero-shot learning transfers through the shared text-image space with no further training.
• Visual prompting: Specifies the target with an example image or box instead of words, which helps for objects that are hard to name, such as a specific part or logo.