Multiple Instance Learning

Multiple instance learning (MIL) is a form of machine learning in which training examples come in groups called bags, and only each bag carries a label, while the individual instances inside it do not. A digitized pathology slide labeled "cancer" is a bag of thousands of tissue tiles, of which perhaps a handful show tumor cells; a surveillance video labeled "incident" is a bag of short clips, most of them uneventful. MIL learns to predict bag labels from such data and, along the way, which instances are responsible for them. It is a kind of weak supervision, used when labeling every instance would take specialists too long or is impossible.

Thomas Dietterich, Richard Lathrop, and Tomás Lozano-Pérez introduced the setting in 1997 for drug activity prediction. A molecule can fold into many shapes, and it binds to a target if at least one of them fits, so they treated each molecule as a bag of its possible shapes, with only the molecule's measured activity as the label.

The Standard MIL Assumption#

The classic assumption for binary labels is that a bag is positive if it contains at least one positive instance and negative only if all of its instances are negative. Copying a bag's label onto every instance, as ordinary supervised learning would, mislabels the many negative instances in each positive bag. MIL instead scores every instance and pools the scores into one bag prediction. Taking the maximum matches the standard assumption exactly: a bag scores as high as its most suspicious instance, and only that instance's score receives the backpropagation signal from the bag's label.

This PyTorch example trains a linear instance scorer on 16,000 bags of 20 random instances, where each positive bag hides one "witness" instance that differs in 4 of its 16 features. Training uses bag labels only, and 1,000 test bags check both the bag predictions and whether the top-scoring instance is the witness:

import torch

torch.manual_seed(0)
n = 17000  # 16,000 training bags and 1,000 test bags, each holding 20 instances with 16 features
bags, labels, witness = torch.randn(n, 20, 16), torch.randint(0, 2, (n,)).float(), torch.randint(0, 20, (n,))
bags[torch.arange(n), witness, :4] += 2.0 * labels[:, None]  # a positive bag hides one shifted "witness" instance

scorer = torch.nn.Linear(16, 1)  # gives every instance a score
optimizer = torch.optim.Adam(scorer.parameters(), lr=1e-2)
for i in range(0, 16000, 32):  # train on 32 bags per step, using bag labels only
    bag_logits = scorer(bags[i : i + 32]).squeeze(-1).max(dim=1).values  # max pooling: a bag scores as its top instance
    loss = torch.nn.functional.binary_cross_entropy_with_logits(bag_logits, labels[i : i + 32])
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

with torch.no_grad():
    scores = scorer(bags[16000:]).squeeze(-1)  # one score per test instance
positive = labels[16000:].bool()
bag_accuracy = ((scores.max(dim=1).values > 0) == positive).float().mean()
located = (scores.argmax(dim=1) == witness[16000:])[positive].float().mean()
print(f"Bag accuracy: {bag_accuracy:.3f}, witness ranked first in {located:.3f} of positive bags")

The model classifies 0.917 of the test bags correctly and ranks the witness first in 0.960 of the positive bags, although no instance label was ever used. That instance ranking is what lets MIL systems point to the tiles or clips behind a decision.

Pooling and Attention-Based MIL#

Max pooling relies on a single instance, so one noisy instance can flip a bag. Mean pooling uses every instance but dilutes a rare signal, and it fits collective assumptions, where a label depends on the proportion of instances showing something. Maximilian Ilse, Jakub Tomczak, and Max Welling proposed attention-based deep MIL in 2018: a small network gives each instance a weight, and the bag representation is the weighted average of the instance features. The attention mechanism handles bags of any size, ignores instance order, and produces weights that serve as a heatmap of the regions behind a prediction, much like saliency maps.

Whole-slide images are too large to train end to end, so many pathology pipelines first convert each tile into a feature vector with a frozen pretrained encoder and train only the pooling network and classifier on slide labels. A YOLO26 image classification model can play that encoder role: model.embed() returns one pooled feature vector per input image, so a batch of tiles becomes a bag of instance vectors for an attention-pooling head.

Applications#

In medical image analysis, MIL removes the need for pathologists to outline tumors on gigapixel slides. Gabriele Campanella and colleagues at Memorial Sloan Kettering trained a MIL system on 44,732 whole-slide images from 15,187 patients using only the diagnoses in pathology reports, and their 2019 study reported an area under the ROC curve above 0.98 for prostate cancer, basal cell carcinoma, and breast cancer metastases to lymph nodes.

In video, Waqas Sultani, Chen Chen, and Mubarak Shah treated each surveillance video as a bag of segments for anomaly detection and released the UCF-Crime dataset of 1,900 untrimmed videos with only video-level labels, which train a model to give the highest anomaly score to the segments where an incident happens. Weakly supervised object detection treats an image as a bag of candidate regions labeled only with the classes present, and the regions that score highest become the predicted bounding boxes. In every one of these settings, keeping all tiles or frames from one slide, patient, or video in the same split avoids the data leakage that would come from scattering near-identical instances across training and test sets.

MIL and Related Settings#

Semi-supervised learning mixes fully labeled and unlabeled examples, while every MIL bag is labeled but none of its instances are. Multi-label classification assigns several classes to one example, and the two combine in multi-instance multi-label learning. Active learning can follow MIL, sending the highest-scoring instances of positive bags to experts when instance-level labels become worth their cost.