Latent Space
A latent space is the compressed space of learned features where a neural network places similar inputs near each other, used for search and generation.
A latent space is the space of internal representations a machine learning model learns, where each input becomes a point, usually with far fewer dimensions than the raw data, and inputs that are similar in meaning sit close together. "Latent" means hidden: the coordinates are not measured in the data but inferred by the model during training. A 640 x 640 color photo holds 1,228,800 pixel values, an image model can summarize it as a vector of a few hundred numbers, and in a well-trained space the nearest neighbors of a photo of a car are other cars, while directions in the space correspond to properties such as pose, color, or style.
Latent spaces work because natural data is highly redundant. Most possible pixel arrays look like noise, and real images occupy a thin, curved region of pixel space, an idea known as the manifold hypothesis. A neural network that learns coordinates along that region can describe each image with a handful of numbers.
Latent Space, Embeddings, and Feature Maps#
An embedding is a single point in a latent space, the vector for one image, word, or user, and the latent space is the set of all such points together with the distances between them; "embedding space" is a synonym. Feature maps inside a convolutional network are spatial latent representations, with a vector at every location, which a model often pools into one embedding per image. Dimensionality reduction methods also produce latent spaces: principal component analysis (PCA) finds a linear one, and t-SNE is mostly used to project a learned latent space down to two dimensions for plotting.
How Models Learn a Latent Space#
An autoencoder learns one directly. Its encoder compresses each input into a short vector, its decoder reconstructs the input from that vector, and minimizing the reconstruction error forces the bottleneck to keep what distinguishes one input from another. No labels are involved, yet the resulting space groups similar inputs, which a k-nearest neighbors (KNN) classifier can measure:
import torch
from sklearn.datasets import load_digits
from sklearn.neighbors import KNeighborsClassifier
from torch import nn
torch.manual_seed(0)
images, labels = load_digits(return_X_y=True) # 1,797 handwritten digits, 8 x 8 = 64 pixels each
x = torch.tensor(images / 16, dtype=torch.float32)
# An autoencoder squeezes 64 pixels through an 8-number bottleneck: the latent space
encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 8))
decoder = nn.Sequential(nn.Linear(8, 32), nn.ReLU(), nn.Linear(32, 64))
optimizer = torch.optim.Adam([*encoder.parameters(), *decoder.parameters()], lr=0.01)
for _ in range(1000):
loss = (decoder(encoder(x)) - x).square().mean() # reconstruction error, no labels used
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Classify held-out digits by their nearest neighbors in each space
codes = encoder(x).detach().numpy()
for name, features in (("64 pixels", images), ("8 latent values", codes)):
knn = KNeighborsClassifier().fit(features[:1400], labels[:1400])
print(f"{name}: {knn.score(features[1400:], labels[1400:]):.2f} accuracy")It prints 64 pixels: 0.97 accuracy and 8 latent values: 0.93 accuracy. Eight learned numbers per image keep enough of the digit's identity for nearest-neighbor classification to stay close to what all 64 pixels achieve, although the autoencoder never saw a label.
Variational autoencoders, introduced by Diederik Kingma and Max Welling in 2013, encode each input as a distribution instead of a point, which makes the space smooth enough to sample new data from it and to interpolate between examples. Contrastive learning shapes a space by pulling matching pairs together and pushing other pairs apart; CLIP trains on image-caption pairs so that images and text share one latent space. Supervised networks learn a latent space as a side effect of their task. model.embed() in Ultralytics YOLO26 runs predict mode and returns one vector per image by averaging the feature map of the layer before the detection head over all spatial positions, a representation suited to clustering and similarity search.
What Latent Spaces Enable#
Generation is cheaper in a compact space. Latent diffusion models run the denoising process of diffusion models on latents instead of pixels. Stable Diffusion, built on the latent diffusion paper, uses an autoencoder to compress a 512 x 512 RGB image into a 64 x 64 x 4 latent, 48 times fewer values, which makes high-resolution generation affordable. Generative adversarial networks (GANs) map random latent vectors to images, so walking between two latent points morphs one generated face into another.
Search and recognition work by measuring distance. Semantic search systems embed documents or images once, store the vectors in a vector database, and return the items with the highest cosine similarity to a query's embedding. In a shared image-text space such as CLIP's, zero-shot learning classifies an image by the class name whose text embedding lies closest, with no training images for that class. For anomaly detection, an autoencoder trained only on normal parts reconstructs defective parts poorly, and inputs whose embeddings fall far from the normal cluster are flagged for inspection.
Directions in a well-organized space encode attributes. In the word2vec embeddings that Tomas Mikolov and colleagues released in 2013, a staple of natural language processing (NLP), the vector for "king" minus "man" plus "woman" lands nearest to "queen".










