CLIP (Contrastive Language-Image Pre-training)

CLIP (Contrastive Language-Image Pre-training) is a model from OpenAI that learns to match images with their text descriptions. It pairs an image encoder with a text encoder and trains both so that an image and a caption describing it map to nearby vectors in one shared embedding space. Trained on 400 million image-caption pairs collected from the internet, CLIP can classify images into categories given only as text, with no labeled examples of those categories, a capability called zero-shot learning. In the 2021 paper, its best model reached 76.2% zero-shot top-1 accuracy on ImageNet, matching the original fully supervised ResNet-50 without using any of the 1.28 million labeled images that ResNet-50 trained on.

How CLIP Is Trained#

CLIP has two encoders. The image encoder is a ResNet or a Vision Transformer (ViT), and the text encoder is a 12-layer transformer with 63 million parameters in its base size. A linear projection maps each encoder's output into the shared space, where every vector is normalized to unit length.

Training uses contrastive learning over batches of 32,768 image-caption pairs. For a batch of N pairs, CLIP computes the cosine similarity of every image with every caption, giving an N×N matrix whose diagonal holds the true pairs. A softmax cross-entropy loss over each row asks every image to pick its own caption out of N, and a second loss over each column asks every caption to pick its image. A learned temperature scales the similarities before the softmax.

Because the captions are free-form text, CLIP learns from whatever the web describes, including objects, scenes, art styles, text in images, and actions, instead of a fixed list of classes.

Zero-Shot Classification#

To classify an image, CLIP embeds each candidate label as a short sentence such as "a photo of a dog", embeds the image, and picks the label with the highest cosine similarity. The prompt wording matters: the paper reports that the template "A photo of a {label}." alone improved ImageNet accuracy by 1.3%, and averaging the text embeddings of 80 templates added another 3.5%. The Hugging Face Transformers pipeline below wraps each label in the template "This is a photo of {}." and runs the original ViT-B/32 model:

from transformers import pipeline

# Load OpenAI's CLIP ViT-B/32 for zero-shot image classification
classifier = pipeline("zero-shot-image-classification", model="openai/clip-vit-base-patch32")

# The classes are free-form text, chosen at inference time
labels = ["a city bus", "a tram", "a delivery truck", "a bicycle"]
predictions = classifier("https://ultralytics.com/images/bus.jpg", candidate_labels=labels)

# Scores are a softmax over the image's similarity to each caption
for prediction in predictions:
    print(f"{prediction['label']}: {prediction['score']:.3f}")

It prints a city bus: 0.657, a tram: 0.341, a delivery truck: 0.001, and a bicycle: 0.000. The tram, another large passenger vehicle, takes nearly all of the remaining probability, and a new label set needs only a new list of strings.

Real-World Applications#

• Semantic search: Image collections are embedded once and stored in a vector database. A text query such as "red sneakers on a wooden floor" is embedded at search time and matched against them, so images are found by content without manual tags.
• Text-to-image generation: Stable Diffusion 1.x conditions its image generator on the text embeddings of CLIP ViT-L/14, and CLIP similarity is widely used to score how closely a generated image follows its prompt.
• Open-vocabulary detection: The YOLO-World models from Tencent AI Lab, which run in the Ultralytics package, encode the class names passed to set_classes with CLIP's ViT-B/32 text encoder, so a detector trained once can find "red backpack" or "forklift" by name. Google's OWLv2 builds on CLIP's pretrained image and text encoders, and the VLM Leaderboard compares it with other open-vocabulary detectors and VLMs on real-world detection datasets.
• Vision encoders for VLMs: Many vision language models start from a pretrained CLIP image encoder. LLaVA, for example, feeds CLIP ViT-L/14 features into a language model.
• Dataset filtering: Large open image-text datasets such as LAION-5B keep only web pairs whose CLIP image-text similarity passes a threshold, removing captions that do not describe their image.

CLIP vs. Related Approaches#

• Supervised learning classifiers: A standard classifier has one output per training class and cannot score a class it never saw. CLIP's "classifier" is the set of text embeddings, so it changes whenever the labels change.
• SigLIP: Google's variant replaces CLIP's batch-wide softmax with a sigmoid loss that scores each image-text pair independently, which uses less memory and works better at small batch sizes.
• Self-supervised contrastive learning: Methods such as SimCLR contrast two augmented views of the same image and learn from images alone. CLIP contrasts across two modalities, which ties its image features to words.
• Generative VLMs: CLIP scores image-text similarity and cannot write a caption or answer a question. Generative models attach a language model to an image encoder, often CLIP's own.

Limitations#

CLIP's zero-shot accuracy drops on fine-grained categories such as aircraft or car models, on counting objects, and on abstract tasks such as estimating distances. It also learns the biases of web captions. Because CLIP learned to read text in images, it can be fooled by it: in OpenAI's multimodal neurons research, an apple with a handwritten label reading "iPod" was classified as an iPod.