Self-Supervised Learning
Self-supervised learning (SSL) trains models on tasks whose labels come from the data itself, such as filling in masked parts. Covers methods and uses.
Self-supervised learning (SSL) trains a model on a task whose labels come from the data itself, such as predicting the next word of a sentence, filling in hidden image patches, or recognizing two crops of the same photo as a match, so it can learn from large amounts of unlabeled data. Supervised learning is limited by data labeling, which is slow and expensive, while raw images, video, audio, and text are plentiful. SSL uses them to pretrain a general-purpose encoder, and fine-tuning or a small trained head then adapts it to a downstream task such as image classification or object detection with far fewer labels. Every modern large language model (LLM) is pretrained this way, by next-token prediction over text as in GPT models, and Meta's DINOv3 scaled self-supervised image pretraining to a 7-billion-parameter model trained on 1.7 billion images.
How Self-Supervised Learning Works#
An SSL method defines a pretext task: it hides or transforms part of each input and trains the network to recover what was removed or to recognize what stayed the same. The pretext task itself is rarely useful. What matters is that solving it forces the network to learn the structure of the data, such as object shapes, textures, and how parts relate, and those features transfer to tasks with labels. After pretraining, the encoder is evaluated or used through linear probing on frozen features or by fine-tuning all of its weights:
Main Families of Methods#
- Masked prediction: Hides part of the input and trains the model to reconstruct it. BERT predicts randomly masked words, and masked autoencoders hide 75% of an image's patches and rebuild their pixels. Predicting the next token of a sequence applies the same idea to text.
- Contrastive learning: Creates two augmented views of each image and trains embeddings so that views of the same image match and views of different images do not. SimCLR and MoCo are the best-known examples.
- Self-distillation: A student network learns to match the output of a teacher network, a slowly updated average of the student, on different crops of the same image, with no negative pairs. BYOL and DINO use this approach, and DINOv2 and DINOv3 scale it to curated datasets of over 100 million images.
- Joint-embedding prediction: Predicts the embeddings of hidden regions from the visible ones, so the model learns abstract content without spending capacity on pixel detail. Meta's I-JEPA and V-JEPA apply it to images and video.
The choice shapes the features. Masked reconstruction keeps fine spatial detail and responds well to fine-tuning, while contrastive and self-distillation methods produce features that work well frozen, which is why DINO-family encoders are popular backbones for depth estimation and semantic segmentation.
Self-Supervised Learning in Computer Vision#
Vision transformers and SSL suit each other: masking whole patches maps directly onto a ViT's token inputs, and ViTs benefit from more training data than labeled datasets provide. Many foundation models start from self-supervised encoders. Meta's SAM 2, which runs in the Ultralytics package for promptable image and video segmentation, builds on a Hiera image encoder pretrained as a masked autoencoder before any segmentation training.
Applications#
- Medical imaging: Hospitals hold far more scans than radiologists can annotate. Pretraining on unlabeled X-rays, CT, or MRI scans gives an encoder that later learns tasks such as tumor detection from a small set of expert labels.
- Satellite imagery: Meta trained a DINOv3 variant on 493 million satellite images, and the World Resources Institute uses it to map tree canopy height. In a region of Kenya, it cut the average height error of earlier DINOv2-based maps from 4.1 to 1.2 meters.
- Speech recognition: wav2vec 2.0 pretrained on 53,000 hours of unlabeled audio and, fine-tuned on only 10 minutes of transcribed speech, reached word error rates of 4.8% and 8.2% on the clean and other LibriSpeech test sets.
Self-Supervised Learning vs. Related Terms#
- Unsupervised learning: Learning without human labels in general, including clustering and density estimation. SSL is usually counted as a form of unsupervised learning that is trained like supervised learning, with targets generated from the data.
- Semi-supervised learning: Trains on a small labeled set and a large unlabeled set at the same time. SSL pretrains on unlabeled data alone, and the two combine well: SimCLRv2 fine-tuned a large self-supervised model on 1% of ImageNet's labels and then distilled it using the unlabeled images.
- Transfer learning: Reuses a pretrained model for a new task. SSL is one way to produce that pretrained model without labels.
- Data augmentation: Random crops, color changes, and flips that expand a labeled dataset. In contrastive SSL, the same transformations define which views count as the same image, so they decide what the encoder learns to ignore.










