Contrastive Learning

Contrastive learning trains a model to produce embeddings in which matching pairs of samples, called positives, land close together and non-matching samples, called negatives, land far apart. The pairs can come from the data itself, such as two augmented crops of the same photo, or from naturally paired data, such as an image and its caption, so contrastive learning needs no class labels. It is one of the main families of self-supervised learning methods for images, alongside masked reconstruction and self-distillation. Encoders trained this way produce features that transfer to detection, classification, and retrieval, and the same objective trains vision-language models such as CLIP, which can then classify images by text alone through zero-shot learning.

How Contrastive Learning Works#

A contrastive method has three parts: a way to make positive pairs, an encoder that maps each sample to a normalized embedding, and a loss that compares similarities. In SimCLR, Ting Chen and colleagues at Google created two views of each image with random cropping, color distortion, and blur, encoded both with a shared ResNet followed by a small projection head, and treated the other images in the batch as negatives:

The data augmentation defines what the model learns to ignore. Because two crops of one image must map to nearby points, the encoder learns that position, scale, and color shifts do not change what the image shows, and it keeps the features that distinguish one image from all the others. With a ResNet-50 four times wider than standard, a linear classifier trained on SimCLR's frozen features reached 76.5% top-1 accuracy on ImageNet, matching a supervised ResNet-50.

Contrastive Loss Functions#

• Pairwise contrastive loss: Raia Hadsell, Sumit Chopra, and Yann LeCun's 2006 loss pulls each positive pair together and pushes each negative pair apart until the two are at least a margin m away.
• Triplet loss: Compares an anchor with one positive and one negative and requires the positive to be closer by a margin. Google's FaceNet trained face embeddings with it in 2015.
• InfoNCE: Compares one positive against many negatives at once. With similarities s divided by a temperature τ, the loss for an anchor is −log[exp(s₊/τ) / Σⱼ exp(sⱼ/τ)], a softmax cross-entropy in which the positive is the correct class among all the candidates. Aaron van den Oord and colleagues named it in contrastive predictive coding in 2018, and SimCLR's NT-Xent loss and CLIP's loss are forms of it.
• Sigmoid loss: SigLIP scores every pair as an independent yes-or-no match with a sigmoid, so the loss needs no normalization over the batch.

In PyTorch, InfoNCE reduces to cross-entropy over a similarity matrix whose diagonal holds the positives:

import torch
import torch.nn.functional as F

torch.manual_seed(0)
# Stand-in embeddings for 8 images, a second view of each (row i of both is a positive pair),
# and 8 unrelated images
view1 = F.normalize(torch.randn(8, 128), dim=1)
view2 = F.normalize(view1 + 0.1 * torch.randn(8, 128), dim=1)
unrelated = F.normalize(torch.randn(8, 128), dim=1)

# InfoNCE: cosine similarities divided by a temperature, then cross-entropy where
# each row's positive (the diagonal) is the correct class among all 8 candidates
temperature = 0.1
targets = torch.arange(8)
for name, keys in [("Augmented views", view2), ("Unrelated images", unrelated)]:
    logits = view1 @ keys.T / temperature
    print(f"{name}: loss {F.cross_entropy(logits, targets):.3f}")

It prints Augmented views: loss 0.016 and Unrelated images: loss 2.505. When each row's positive is clearly the most similar key, the loss approaches zero. With unrelated keys it rises above ln 8 ≈ 2.08, the loss of a uniform guess among eight candidates, because dividing by a temperature of 0.1 turns chance similarities into confident wrong guesses. A lower temperature sharpens the softmax and concentrates the loss on the hardest negatives. MoCo uses τ = 0.07, and CLIP initializes its learned temperature to the same value.

Negatives, Batch Size, and Variants#

InfoNCE improves with more negatives, which makes the number of negatives a central cost. SimCLR drew them from very large batches. MoCo, from Kaiming He and colleagues at Facebook AI Research, kept a queue of 65,536 embeddings from recent batches as negatives and encoded them with a slowly updated momentum copy of the encoder, which decouples the number of negatives from the batch size. Random negatives can be false negatives, such as another photo of the same dog breed, and very easy negatives teach little, which is why hard negative mining seeks out confusable samples. Supervised contrastive learning (SupCon) uses labels to treat every sample of the same class as a positive. Non-contrastive methods such as BYOL, SimSiam, and DINO drop negatives entirely and avoid collapsing every input to the same embedding with asymmetric networks, stop-gradients, or centering.

Contrastive Learning vs. Related Concepts#

• Metric learning: The broader task of learning a distance function, which contrastive and triplet losses come from. "Contrastive learning" now usually refers to the self-supervised, many-negative form popularized by SimCLR and MoCo.
• Autoencoders: Learn by reconstructing the input, so their features must keep pixel-level detail. Contrastive features keep only what distinguishes one sample from the others.
• JEPA: Predicts the embeddings of hidden image regions from visible ones, learning in embedding space without negative pairs.
• Supervised learning: Trains toward a fixed set of human-assigned classes, while contrastive pretraining needs only a rule for which samples match.

Applications#

• Vision-language models: CLIP trained an image encoder and a text encoder on 400 million image-caption pairs with batches of 32,768, so text prompts can classify and retrieve images. Ultralytics YOLOE-26 models encode class names with Apple's MobileCLIP2 text encoder, trained with a CLIP-style contrastive objective, and score every candidate region against those text embeddings in a contrastive head, so set_classes(["forklift", "pallet"]) turns the same weights into a detector for new categories.
• Semantic search: Contrastively trained embeddings of images and text are stored in a vector database and searched by similarity.
• Pretraining with few labels: In fields such as medical image analysis, where expert labels are scarce, contrastive pretraining on unlabeled scans gives a backbone that fine-tuning adapts with a small labeled set.
• Face recognition and re-identification: Contrastive and triplet losses train embeddings in which photos of the same person or object sit close together. FaceNet's 128-dimensional embeddings reached 99.63% accuracy on the Labeled Faces in the Wild benchmark.