Dimensionality Reduction

Dimensionality reduction transforms data described by many features into a representation with far fewer features that keeps the information that matters, such as the variance of the data, the distances between points, or which points are neighbors. A 640x640 color image has more than 1.2 million pixel values, a gene-expression profile can have 20,000 measurements, and an image embedding often has hundreds of dimensions. Reducing them speeds up machine learning models, lowers the risk of overfitting, cuts storage, and makes it possible to plot data in two or three dimensions for data visualization.

Why Fewer Dimensions Help#

Richard Bellman coined the phrase "curse of dimensionality" in his work on dynamic programming to describe how problems grow harder as dimensions are added. The volume of the space grows exponentially with each new feature, so a fixed amount of training data covers it ever more sparsely, and in very high dimensions the distances between random points become nearly equal, which weakens methods built on distance such as k-nearest neighbors and k-means clustering.

Real data rarely fills its space. Pictures of handwritten digits vary mainly in stroke thickness, slant, and a few other factors, so they lie close to a much lower-dimensional surface, or manifold, inside pixel space. Dimensionality reduction finds coordinates on that surface, and because it needs no labels, most methods are a form of unsupervised learning.

Linear and Nonlinear Methods#

Principal component analysis (PCA), introduced by Karl Pearson in 1901, rotates the data onto orthogonal directions ordered by how much variance they capture and keeps the first few. Its explained-variance ratio shows how many components a dataset needs. Linear discriminant analysis, from Ronald Fisher in 1936, uses class labels to find the directions that best separate classes. Random projection multiplies the data by a random matrix, and the Johnson-Lindenstrauss lemma guarantees that the distances between n points survive within a chosen tolerance using a number of dimensions that grows only with the logarithm of n.

Nonlinear methods follow curved manifolds. Isomap and locally linear embedding, both published in Science in 2000, preserve distances along the manifold and local linear neighborhoods. t-SNE and UMAP place similar points close together on a 2D map, which makes them the standard tools for viewing clusters of embeddings, although distances between far-apart clusters on those maps carry little meaning. Autoencoders learn an encoder that maps each input to a small latent space code and a decoder that reconstructs it, and in 2006 Geoffrey Hinton and Ruslan Salakhutdinov showed that deep autoencoders give better codes than PCA for images and documents.

This example compresses 8x8 digit images from 64 pixel features to the fewest principal components that keep 90% of the variance, then classifies the digits with k-nearest neighbors before and after:

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline

X, y = load_digits(return_X_y=True)  # 1,797 images of 8x8 pixels = 64 features

# Keep the fewest principal components that explain 90% of the variance
pca = PCA(n_components=0.90).fit(X)
print(f"Features: {X.shape[1]} -> {pca.n_components_}")

# Classify with k-nearest neighbors on the raw pixels and on the reduced features
raw = cross_val_score(KNeighborsClassifier(), X, y).mean()
reduced = cross_val_score(make_pipeline(PCA(n_components=0.90), KNeighborsClassifier()), X, y).mean()
print(f"Accuracy: {raw:.3f} with 64 features, {reduced:.3f} with {pca.n_components_}")

Twenty-one components hold 90% of the variance, and cross-validated accuracy moves only from 0.963 to 0.959 with a third of the features. Fitting PCA inside the pipeline learns the projection from each training fold alone, so no information leaks from the test folds.

Feature Selection and Feature Extraction#

Feature selection keeps a subset of the original columns, such as dropping redundant sensor channels, so the remaining features keep their meaning. Dimensionality reduction in the narrower sense is feature extraction: it builds new features as combinations of the old ones, which usually compresses better but is harder to interpret. Both are steps in data preprocessing and feature engineering. Deep networks perform extraction as part of training, compressing pixels into embeddings, and Matryoshka representation learning trains embeddings whose leading dimensions work on their own, so a vector database can store shorter vectors.

Uses in Computer Vision#

Computer vision relies on reduced representations at several stages. Projecting image embeddings to 2D with t-SNE or UMAP shows whether classes separate and where outliers sit, and PCA or vector quantization shrinks the embeddings behind vector search so millions of images fit in memory. YOLO26 instance segmentation models build the idea into their output: the head computes 32 prototype masks per image at a quarter of the input resolution and predicts only 32 coefficients per object, and each object's mask is the weighted sum of the prototypes, so a 160x160 mask for a 640-pixel input, 25,600 values, is encoded in 32 numbers.