Matryoshka Representation Learning (MRL)
Matryoshka Representation Learning trains embeddings whose leading dimensions work alone, so one vector can be truncated for cheaper storage and search.
Matryoshka Representation Learning (MRL) is a training method that makes the leading dimensions of an embedding usable as an embedding on their own. A model trained with MRL outputs one vector, for example 768 dimensions long, whose first 512, 256, or 128 values can be cut off and used directly for search or classification, with accuracy falling gradually as the vector shrinks. The name comes from Russian nesting dolls. Kusupati et al. introduced Matryoshka Representation Learning at NeurIPS 2022 and reported embeddings up to 14 times smaller at the same ImageNet-1K classification accuracy, plus up to 14 times faster large-scale retrieval.
How MRL Works#
A standard embedding model applies its loss function to the full vector, so information spreads across all dimensions and any truncation throws part of it away. MRL picks a set of nested sizes, typically doubling from 8 up to the full width, applies the task loss to each prefix of the embedding, and sums the losses. Every prefix has to solve the task by itself, which pushes the most useful information into the earliest dimensions. The encoder still runs once per input, so training costs little more than before, and the efficient variant MRL-E shares a single classifier across sizes, with each prefix using the first columns of its weight matrix. The example below trains a standard model and an MRL model on scikit-learn's handwritten digits and classifies the test set from truncated embeddings:
import torch
import torch.nn.functional as F
from sklearn.datasets import load_digits
torch.manual_seed(0)
x, y = load_digits(return_X_y=True)
x, y = torch.tensor(x / 16, dtype=torch.float32), torch.tensor(y)
n, dims = 1400, [2, 4, 8, 16, 64] # train/test split and nested prefixes of a 64-dim embedding
for nested in (False, True):
encoder = torch.nn.Sequential(torch.nn.Linear(64, 256), torch.nn.ReLU(), torch.nn.Linear(256, 64))
head = torch.nn.Linear(64, 10) # shared classifier: its first d columns score a d-dim prefix
optimizer = torch.optim.Adam([*encoder.parameters(), *head.parameters()], lr=1e-3)
sizes = dims if nested else [64] # MRL adds a loss for every prefix, a standard model only for all 64
for step in range(500):
z = encoder(x[:n])
loss = sum(F.cross_entropy(z[:, :d] @ head.weight[:, :d].T + head.bias, y[:n]) for d in sizes)
optimizer.zero_grad()
loss.backward()
optimizer.step()
z = encoder(x[n:]).detach() # test embeddings, classified from each truncated prefix
accuracy = [((z[:, :d] @ head.weight[:, :d].T + head.bias).argmax(1) == y[n:]).float().mean() for d in dims]
print("MRL: " if nested else "Standard:", ", ".join(f"{d}d {a:.2f}" for d, a in zip(dims, accuracy)))Both models reach 0.91 test accuracy with all 64 dimensions. Truncated to 4 dimensions, the standard model falls to 0.43 and the MRL model keeps 0.88; at 2 dimensions the scores are 0.22 and 0.66.
Adaptive Retrieval#
MRL makes vector search cheaper in two ways. Storing only a prefix shrinks a vector database in proportion, and adaptive, or funnel, retrieval searches the whole collection with a short prefix and then rescores a shortlist with the full vectors. Truncated vectors must be normalized to unit length again before computing cosine similarity or dot products. Ultralytics Platform runs embedding search across datasets: its Find similar images action compares per-image embeddings across public, personal, and team datasets and returns up to 24 of the nearest images with their similarity scores, a shared index whose memory and search cost grow with every embedding dimension.
MRL vs. Related Concepts#
- Dimensionality reduction: Methods such as principal component analysis (PCA) fit a projection to a fixed set of vectors after training, and the projection has to be stored and applied to every new vector. MRL builds the ordering into training, so truncation is a slice.
- Vector quantization: Stores each vector with fewer bits instead of fewer dimensions. The two combine, for example binary-quantized MRL prefixes for a first-pass search.
- Model pruning and knowledge distillation: Shrink the model that produces embeddings. MRL leaves the encoder unchanged and shrinks only its output.
- Reranker: A separate, more expensive model that rescores a shortlist. Adaptive retrieval gets a cheaper version of the same two-stage pattern from a single MRL embedding.
MRL in Embedding Models#
OpenAI's text-embedding-3 models, released in January 2024, support shortening embeddings through a dimensions parameter, and text-embedding-3-large cut to 256 dimensions still outperforms the 1,536-dimension text-embedding-ada-002 on the MTEB benchmark. Sentence Transformers trains MRL models with its MatryoshkaLoss, covered in Hugging Face's introduction to Matryoshka embedding models. Google's EmbeddingGemma, a text embedding model with about 300 million parameters, outputs 768 dimensions and offers 512, 256, and 128 through MRL truncation.










