Principal Component Analysis (PCA)

Principal component analysis (PCA) is a linear dimensionality reduction technique that re-expresses data as new, uncorrelated variables called principal components, ordered by how much of the data's variance each one captures. Keeping only the first few components compresses many correlated features into a handful while preserving most of the information. Karl Pearson introduced the method in 1901, and Harold Hotelling developed it further and named it in 1933. PCA is an unsupervised technique that ignores labels, and it serves as a standard data preprocessing step, a fast way to visualize high-dimensional data in two or three dimensions, and a baseline for compression and noise removal in machine learning.

How PCA Works#

PCA first centers each feature by subtracting its mean. The first principal component is the direction along which the centered data varies most, and each later component is the direction of greatest remaining variance that is orthogonal to all earlier ones. The components are the eigenvectors of the data's covariance matrix, and the matching eigenvalues are the variances along them; libraries compute both with a singular value decomposition (SVD) of the centered data, which is more numerically stable than forming the covariance matrix. Projecting the data onto the top k components gives the reduced representation, and mapping it back reconstructs an approximation of the original. Among all linear projections onto k dimensions, PCA's gives the smallest squared reconstruction error.

Variance depends on units, so features measured on different scales, such as income in dollars and age in years, are standardized first; otherwise the feature with the largest numeric range dominates the first component. The number of components k is usually chosen from the explained variance ratio, keeping components until their cumulative share of the variance reaches a target such as 95%, or from the elbow of a scree plot of the variance per component.

Scikit-learn's PCA accepts a variance target directly. This example compresses 8×8 images of handwritten digits:

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

# 1,797 handwritten digits, each an 8x8 image flattened to 64 pixel values
X, _ = load_digits(return_X_y=True)

# Keep the fewest components that explain 95% of the variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)

print(f"{X.shape[1]} features -> {pca.n_components_} components")
print(f"First two components: {pca.explained_variance_ratio_[:2].sum():.1%} of the variance")

Twenty-nine components keep 95% of the variance in the 64 pixels, cutting the input size by more than half, and the first two components alone hold 28.5%. The variance concentrates because neighboring pixels are strongly correlated and the pixels along the left and right edges of the images barely vary.

Limitations#

PCA finds only linear structure, so data lying on a curved surface needs more components than its true dimensionality. Each component mixes every original feature, which makes components hard to interpret, and squared distances let a few outliers tilt them. High variance also differs from usefulness: a low-variance direction can carry the signal that separates two classes, so discarding it can hurt a classifier even though little variance is lost. Reducing many features to a few still helps with the curse of dimensionality, where models need far more samples to generalize as the number of features grows, and it limits overfitting when features are many and samples few.

PCA for Embeddings in Computer Vision#

Vision models describe each image or object with an embedding, a vector of hundreds of numbers, and PCA is a common first step for plotting, clustering, or compressing those vectors. YOLO26 models return one embedding per image through model.embed(), and projecting those vectors onto their first two principal components gives a quick linear map of how a custom-trained model separates its classes. Reducing stored embeddings to fewer dimensions also shrinks the index of a vector database and speeds up vector search.

Applications#

• Facial recognition: The eigenfaces method of Turk and Pentland (1991) represented each face by its coordinates on the principal components of a set of face images and recognized people by comparing those coordinates.
• Anomaly detection: Industrial process monitoring fits PCA to sensor data from normal operation and raises an alarm when a new reading reconstructs poorly or falls far out along the kept components.
• Population genetics: PCA of genotype data summarizes ancestry, and in a 2008 study of 1,387 Europeans the first two components reproduced a recognizable map of Europe.
• Interest rate risk: Changes in a bond yield curve are dominated by three components that analysts call level, slope, and curvature, so risk models track three numbers instead of every maturity.

PCA vs. Related Techniques#

• t-SNE and UMAP: Nonlinear methods built for visualization that preserve local neighborhoods, so clusters separate more clearly, while distances between far-apart clusters lose meaning. Scikit-learn's t-SNE has no transform for new points, while a fitted PCA projects new data with one matrix multiplication.
• Autoencoders: Neural networks that compress and reconstruct data and can learn nonlinear structure. A linear autoencoder trained on squared error learns the same subspace as PCA.
• Feature selection and extraction: Feature selection keeps a subset of the original columns, which stay interpretable, while PCA, a classic feature extraction method, builds new features that combine all of them.
• Linear discriminant analysis (LDA): A supervised relative that chooses directions maximizing the separation between classes instead of total variance, and yields at most one fewer component than the number of classes.