Capsule Networks (CapsNet)

Capsule networks (CapsNets) are neural networks built from capsules, small groups of neurons whose output is a vector instead of a single number. The length of a capsule's vector is the probability that an entity, such as a digit, an eye, or a wheel, is present, and its direction encodes that entity's instantiation parameters: position, size, orientation, stroke thickness, and other properties of pose and deformation. Geoffrey Hinton, Alex Krizhevsky, and Sida Wang introduced capsules in a 2011 paper on transforming autoencoders, and Sara Sabour, Nicholas Frosst, and Hinton turned them into a working classifier in Dynamic Routing Between Capsules, presented at NIPS 2017, which connects capsules in successive layers with an iterative algorithm called routing by agreement.

Why Capsules Replace Max Pooling#

A convolutional neural network (CNN) detects features with scalar activations and often shrinks its feature maps with max pooling, which keeps the strongest response in each window and discards where in the window it occurred. That makes detection tolerant to small shifts, but a network that only checks whether parts are present can accept a face with the mouth above the eyes, the so-called Picasso problem. Hinton argued for equivariance over invariance: when an object moves or rotates, a capsule's vector should rotate with it while its length, the probability that the object exists, stays the same. Capsules that learn viewpoint this way need less of the data augmentation a CNN uses to recognize objects from new angles.

Ultralytics YOLO26 keeps object positions without capsules: five stride-2 convolutions reduce its input to 1/32 resolution, and its only max-pooling block, SPPF, pools at stride 1 with a 5 x 5 window three times in a row, widening the receptive field without shrinking the grid its detection head predicts from.

How Dynamic Routing Works#

Each capsule in a lower layer multiplies its output by a learned weight matrix to predict the output of every capsule in the layer above. Routing then decides how much of each child's output reaches each parent. A softmax over routing logits, which start at zero, gives coupling coefficients that split every child's output across the parents. Each parent sums the predictions weighted by those coefficients and applies the squash function, which keeps the vector's direction and maps its length into the range 0 to 1. The dot product between each prediction and the parent's new output is added to the logit, so children send more of their output to parents whose state they predicted:

import torch

torch.manual_seed(0)

# 6 child capsules each predict an 8D pose for 2 parent capsules
predictions = 0.2 * torch.randn(6, 2, 8)
pose = 0.5 * torch.randn(8)
predictions[:4, 0] = pose + 0.05 * torch.randn(4, 8)  # children 0-3 agree on parent 0

logits = torch.zeros(6, 2)
for _ in range(3):  # routing by agreement, 3 iterations as in the paper
    coupling = logits.softmax(dim=1)  # each child splits its vote across parents
    s = (coupling.unsqueeze(-1) * predictions).sum(dim=0)
    squared = s.square().sum(dim=-1, keepdim=True)
    parents = squared / (1 + squared) * s / squared.sqrt()  # squash: length in [0, 1)
    logits = logits + (predictions * parents).sum(dim=-1)  # agreement raises the logit

print(coupling[:, 0].round(decimals=2))
print(parents.norm(dim=-1).round(decimals=2))

It prints tensor([0.9700, 0.9700, 0.9700, 0.9700, 0.5700, 0.4700]) and tensor([0.9800, 0.1500]). After three iterations, the four children that agree send 97% of their output to parent 0, the two others stay close to an even split, and parent 0's vector length reaches 0.98 while parent 1, which only received scattered votes, stays at 0.15. Routing resembles the attention mechanism of a transformer run in the opposite direction: the softmax runs over the parents a child could feed, so parents compete for each child's output.

The 2017 CapsNet and Its Results#

The original network is shallow: a convolution with 256 kernels of size 9 x 9 feeds PrimaryCaps, a convolutional capsule layer that turns a 28 x 28 digit into 32 x 6 x 6 = 1,152 eight-dimensional capsules, and routing connects them to DigitCaps, one 16-dimensional capsule per class. Training combines a margin loss function on capsule lengths, which pushes the length above 0.9 for the digits present and below 0.1 for the rest, with a decoder that reconstructs the image from the correct digit capsule, an autoencoder-style regularization term scaled down by 0.0005. Perturbing one of the 16 dimensions and decoding the result changes a single property such as scale, stroke thickness, skew, or width, which makes capsule outputs easier to inspect than CNN activations, a goal shared with explainable AI.

With 8.2 million parameters against 35.4 million for a baseline CNN, CapsNet reached 0.25% test error on MNIST, compared with 0.39% for the baseline, and 5.2% against 8.1% on MultiMNIST, pairs of digits whose bounding boxes overlap by 80% on average. Trained only on MNIST digits shifted by up to 2 pixels, it scored 79% on affNIST, the same digits under random affine transformations, while a CNN with a similar parameter count scored 66%. On CIFAR-10 an ensemble of seven CapsNets reached 10.6% error, about what CNNs achieved when first applied to that dataset, and the authors traced the gap to capsules trying to account for cluttered backgrounds.

Limits and Later Capsule Models#

Routing runs several iterations over every pair of child and parent capsules, so compute and memory grow quickly with image size and class count, and capsule networks never matched CNNs or vision transformers on large image classification benchmarks such as ImageNet. Researchers have applied them in medical image analysis, for example to classify brain tumor types from MRI scans, where training sets are small.

Matrix capsules with EM routing, by Hinton, Sabour, and Frosst at ICLR 2018, replaced pose vectors with 4 x 4 pose matrices and routed with expectation-maximization; on the smallNORB benchmark of toy objects photographed from many viewpoints, it cut test errors by 45% relative to the previous state of the art and resisted white-box adversarial attacks better than a baseline CNN. Stacked capsule autoencoders followed in 2019 and learned capsules without labels, and Hinton's 2021 GLOM proposal represents the same part-whole hierarchies as islands of agreeing embedding vectors, with no dynamic routing at all.