Memory Bank

A memory bank is a store of feature vectors, or embeddings, that a model computed earlier, kept so that new inputs can be compared with far more examples than fit in one batch or one forward pass. Depending on the method, the stored vectors come from previous training batches, previous video frames, or a reference set of normal images, and the bank is read with similarity lookups or attention and updated as new vectors arrive. Memory banks are central to three areas of computer vision: contrastive learning, where they supply thousands of negative examples per step; video segmentation and tracking, where they remember how an object looked in earlier frames; and anomaly detection, where they hold what normal data looks like.

Memory Banks in Contrastive Learning#

Contrastive self-supervised learning trains an encoder to pull two views of the same image together and push other images away, and more negatives generally give a better training signal. Comparing against more images normally requires a larger batch size, which GPU memory limits. Zhirong Wu and colleagues introduced the memory bank for this purpose in Unsupervised Feature Learning via Non-Parametric Instance Discrimination at CVPR 2018: it kept a 128-dimensional feature for every one of the 1.28 million ImageNet training images, refreshed each entry whenever its image was seen again, and drew thousands of negatives from it per step. Because entries were written by older versions of the encoder, the stored features drifted out of date and became inconsistent with the current model.

Momentum Contrast (MoCo), by Kaiming He and colleagues at Facebook AI Research, fixed this with a queue and a second encoder. The bank holds the keys of the most recent batches, 65,536 by default, enqueuing each new batch and dropping the oldest. The keys come from a momentum encoder whose weights follow the main encoder slowly, updated as 0.999 times their old value plus 0.001 times the main encoder's weights, so keys written at different steps stay consistent with each other.

Memory Banks in Video Segmentation and Tracking#

In video, a memory bank lets a model handle occlusion and appearance change by consulting earlier frames. Space-Time Memory networks (STM) stored key and value features of past frames with their masks and segmented each new frame by attending to them, and XMem extended this to long videos with separate short-term and long-term memories. Meta's SAM 2 keeps a memory bank of the frames a user prompted plus the most recent frames, and its memory attention conditions every new frame's features on them, which is how a mask follows an object that leaves and re-enters the view. In the Ultralytics package, SAM 2 attends to the prompted frames plus the 6 most recent ones by default, and SAM2DynamicInteractivePredictor adds a prompted image's object state to its memory bank when called with update_memory=True, then segments later images against what it stored.

Multi-object trackers use the same idea for re-identification: DeepSORT keeps a gallery of each track's last 100 appearance descriptors and matches new detections to the track whose stored descriptors are closest.

Memory Banks for Anomaly Detection#

PatchCore, presented by Karsten Roth and colleagues at CVPR 2022, builds a memory bank of patch features that an ImageNet-pretrained CNN extracts from defect-free product images. A test patch is scored by its distance to the nearest stored feature, and the image score is the worst patch score, so anything unlike all normal patches stands out. Greedy coreset subsampling keeps only a small fraction of the features while still covering their spread, and the method reached an image-level AUROC of up to 99.6% on the MVTec AD industrial defect detection benchmark.

This PyTorch example applies the same recipe to 8x8 handwritten digits, with raw pixels as features: 100 images of the digit 3 form the memory bank, and every other digit counts as an anomaly:

import torch
from sklearn.datasets import load_digits
from sklearn.metrics import roc_auc_score

X, y = load_digits(return_X_y=True)
X, y = torch.tensor(X / 16, dtype=torch.float32), torch.tensor(y)
normal = X[y == 3]  # images of the digit 3 play the role of defect-free parts
memory_bank = normal[:100]  # memory bank: stored features of 100 normal images
test = torch.cat([normal[100:], X[y != 3]])  # unseen 3s plus every other digit as anomalies
is_anomaly = torch.cat([torch.zeros(len(normal) - 100), torch.ones(len(X) - len(normal))])

# Greedy coreset: repeatedly keep the stored feature farthest from everything kept so far
coreset = [0]
distance = torch.cdist(memory_bank, memory_bank[:1])[:, 0]
for _ in range(9):
    coreset.append(int(distance.argmax()))
    distance = torch.minimum(distance, torch.cdist(memory_bank, memory_bank[coreset[-1:]])[:, 0])

for name, bank in [("Full memory bank, 100 entries", memory_bank), ("Coreset, 10 entries", memory_bank[coreset])]:
    scores = torch.cdist(test, bank).min(dim=1).values  # anomaly score: distance to the nearest stored feature
    print(f"{name}: AUROC {roc_auc_score(is_anomaly, scores):.3f}")

On the 1,697 test images, the full bank separates the 83 unseen 3s from the other digits with an area under the ROC curve (AUC) of 0.984, and the 10-entry coreset scores 0.982, keeping almost all of the detection quality with a tenth of the memory and lookups.

Memory Bank vs. Related Concepts#

• Vector database: Stores embeddings persistently and serves similarity search to many applications. A memory bank lives inside one model's training or inference loop, usually in GPU memory, and is overwritten as the model runs.
• Context window: The tokens a transformer attends to in one pass. A memory bank holds vectors outside that window, from other batches, frames, or images.
• Transformer-XL: Caches the hidden states of the previous text segment as a memory that the next segment attends to, a memory bank of activations that extends a language model's reach past one window.
• Hard negative mining: Picks the most confusing negatives for training. A memory bank supplies a large pool of negatives that such selection can draw from.
• Episodic memory: In AI agents and reinforcement learning, a store of past experiences retrieved to guide decisions. Memory banks store vectors for a model's own computation, such as negatives, past frames, or normal patches.