Masked Autoencoders (MAE)

Masked autoencoders (MAE) are a self-supervised learning method for images: the model hides a large random share of an image's patches, typically 75%, and learns to reconstruct the missing pixels from the visible rest. Kaiming He and colleagues at Meta AI introduced the method in the 2021 paper Masked Autoencoders Are Scalable Vision Learners, adapting the masked word prediction of BERT to vision transformers. Because the encoder processes only the visible patches, MAE pretraining ran 3 times or more faster than processing every patch, and a ViT-Huge model pretrained this way reached 87.8% accuracy on ImageNet, the best result at the time among methods that used only ImageNet-1K data. The pretrained encoder then serves as a backbone for classification, detection, and segmentation.

How Masked Autoencoders Work#

MAE splits an image into a grid of non-overlapping patches, 196 patches of 16x16 pixels for a 224x224 input, and randomly hides most of them. Its encoder and decoder are deliberately unequal. The encoder, a standard ViT, runs only on the visible patches, which cuts its compute and memory to about a quarter. A lightweight decoder, by default 8 transformer blocks with under 10% of the encoder's computation per token, receives the encoded visible patches plus a shared learnable mask token for every hidden position and predicts the pixel values of each hidden patch. The loss function is the mean squared error on the hidden patches only, and using per-patch normalized pixels as targets improved the learned features further. After pretraining, the decoder is discarded:

The high masking ratio is what makes the task useful. Neighboring image patches are highly redundant, so with only a few patches hidden, the model can fill each gap by interpolating from its neighbors. With 75% hidden, it has to infer whole objects and scenes from fragments, which forces it to learn shapes, parts, and context. BERT, by contrast, masks only 15% of words, because each word already carries dense meaning.

The core of MAE is its masking step, shown here in PyTorch on a batch of four images:

import torch

torch.manual_seed(0)
images = torch.rand(4, 3, 224, 224)  # a batch of 4 RGB images

# Patchify: each 224x224 image becomes 196 patches of 16x16 pixels, flattened to 768 values
patches = images.unfold(2, 16, 16).unfold(3, 16, 16).permute(0, 2, 3, 1, 4, 5).reshape(4, 196, 768)

# Random masking as in the MAE reference code: rank patches by random noise and keep the first 25%
keep = int(196 * (1 - 0.75))
ids_keep = torch.argsort(torch.rand(4, 196), dim=1)[:, :keep]
visible = torch.gather(patches, 1, ids_keep.unsqueeze(-1).expand(-1, -1, 768))

# Mask of hidden patches (1 = hidden): the decoder predicts these, and the loss is averaged over them only
mask = torch.ones(4, 196).scatter(1, ids_keep, 0)
print(f"Encoder input: {visible.shape[1]} of {patches.shape[1]} patches per image")
print(f"Reconstruction targets: {int(mask[0].sum())} hidden patches per image")

It prints Encoder input: 49 of 196 patches per image and Reconstruction targets: 147 hidden patches per image. The encoder's transformer layers therefore process a sequence one quarter the length of the full image, and because attention compares every token with every other, each attention score matrix is one sixteenth the size it would be for all 196 patches.

Results and Variants#

MAE features are strongest after fine-tuning. Under linear probing, a ViT-L encoder pretrained with MAE reached 73.5% ImageNet accuracy, below contrastive methods, but when fine-tuned and transferred to COCO object detection and segmentation, MAE pretraining outperformed supervised pretraining. The recipe has been adapted well beyond still images: VideoMAE masks 90% or more of the space-time patches of a video clip, SatMAE pretrains on satellite imagery, and ConvNeXt V2 brings masked autoencoding to convolutional networks with sparse convolutions.

Masked pretraining also sits under widely used foundation models. Meta's Segment Anything Model (SAM), which runs in the Ultralytics package, builds its image encoder on an MAE-pretrained ViT, and its ViT-B version, sam_b.pt, is the default mask model in Ultralytics auto_annotate, which turns the boxes of a YOLO26 detector into segmentation labels.

Masked Autoencoders vs. Related Terms#

• Autoencoder: Compresses the whole input into a small code and reconstructs it. An MAE sees an incomplete input and must predict the parts it never saw, and its encoder output keeps one vector per visible patch.
• Masked image modeling: The broader family MAE belongs to. BEiT predicts discrete visual tokens for masked patches, and SimMIM feeds mask tokens through the encoder, while MAE predicts pixels and leaves mask tokens out of the encoder.
• Contrastive learning: Learns by matching augmented views of the same image and needs no decoder, producing features that work better frozen, while MAE features improve more with fine-tuning.
• Joint Embedding Predictive Architecture (JEPA): Also hides image regions, but predicts their embeddings instead of their pixels, which keeps the model from spending capacity on texture detail.