Normalization

Normalization is the rescaling of numeric values to a common scale so that no feature dominates a model just because of its units. In data preprocessing, it usually means transforming each input feature to a fixed range such as [0, 1], or to zero mean and unit variance, before training. Inside a neural network, normalization layers such as batch normalization apply the same idea to intermediate activations. Without it, a feature measured in thousands, such as income, outweighs a feature measured in single digits, such as household size, in every distance calculation and gradient step.

Common Normalization Techniques#

• Min-max scaling: Maps each feature to [0, 1] with (x − min) / (max − min). One extreme value compresses all the others into a narrow band, so it suits features with known, fixed bounds, such as 8-bit pixel values divided by 255.
• Z-score standardization: Subtracts the feature's mean and divides by its standard deviation, giving mean 0 and standard deviation 1. It is the usual default for features with no fixed bounds.
• Robust scaling: Subtracts the median and divides by the interquartile range, so a few outliers barely change how the remaining values are scaled.
• Log scaling: Applies a logarithm, often log(1 + x), to compress heavy-tailed features such as prices, populations, or view counts before other scaling.
• Unit-vector (L2) normalization: Divides each sample's vector by its length so that every vector has norm 1. Applied to embeddings, it turns cosine similarity into a plain dot product. scikit-learn's Normalizer performs this per-sample operation, which surprises users who expect it to scale each feature like MinMaxScaler.

Which technique fits depends on what is being scaled and how its values are distributed:

Why Normalization Matters#

Distance-based methods such as k-nearest neighbors (k-NN), k-means clustering, and RBF-kernel support vector machines compare raw Euclidean distances, so the widest-ranging feature decides which points count as close. Gradient-based models train faster on normalized inputs, because features on very different scales stretch the loss surface into narrow valleys where gradient descent must take small steps. Tree ensembles such as random forests split on one feature threshold at a time and give the same splits under any monotonic rescaling. Scaling the Wine dataset before k-NN shows the size of the effect:

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler, StandardScaler

# 13 chemical features on very different scales, from hue (about 0.5-1.7) to proline (about 280-1,680)
X, y = load_wine(return_X_y=True)

# k-NN compares raw distances, so unscaled proline dominates every distance
for name, scaler in [("none", "passthrough"), ("min-max", MinMaxScaler()), ("z-score", StandardScaler())]:
    # The pipeline fits the scaler on each training fold only, so no test statistics leak in
    model = make_pipeline(scaler, KNeighborsClassifier())
    accuracy = cross_val_score(model, X, y, cv=5).mean()
    print(f"scaling {name:8s} 5-fold accuracy: {accuracy:.3f}")

Without scaling, k-NN reaches 0.691 accuracy, because proline, which spans about 1,400 units, outweighs the other 12 features in every distance. Min-max scaling raises accuracy to 0.950 and z-score standardization to 0.949. Fitting the scaler inside the pipeline matters: computing the minimum, maximum, mean, or standard deviation on the full dataset before splitting leaks test-set information into training, a form of data leakage.

Normalization in Computer Vision#

Images are normalized per channel. Dividing 8-bit pixel values by 255 maps them to [0, 1], and many classification backbones then subtract a per-channel mean and divide by a standard deviation: torchvision's ImageNet-pretrained ResNet-50 expects mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225). Feeding a pretrained model inputs normalized differently from its training data shifts every activation and lowers accuracy. Ultralytics YOLO preprocessing divides pixel values by 255 and applies no mean or standard-deviation shift, in training and in prediction, so YOLO26 always receives inputs in [0, 1].

Medical images need modality-specific rules. CT scans store intensities in Hounsfield units, a physical scale on which air is −1000 and water is 0, so one set of statistics works across scans: the widely used nnU-Net segmentation framework clips CT intensities to the 0.5th and 99.5th percentiles of foreground values across all training cases, then z-scores them with the dataset-wide mean and standard deviation. MRI intensities have no fixed physical scale and vary between scanners, so nnU-Net z-scores each MRI scan on its own.

Normalization vs. Related Terms#

• Standardization: The usual name for z-score scaling. Many sources call both min-max and z-score scaling "normalization," while others reserve "normalization" for min-max scaling.
• Data augmentation: Changes image content or geometry, such as flips, crops, and color shifts, to add variety, while normalization only changes the scale of values.
• Normalization layers: Batch normalization standardizes each channel with statistics from the current mini-batch. Layer normalization standardizes each example across its features, independent of batch size, and is used throughout transformers. These layers follow normalization with a learned scale and shift.
• RMSNorm: A simplified layer normalization that skips mean subtraction and divides by the root mean square alone. It is the normalization layer in Meta's Llama and many other large language models.