Model Pruning

Model pruning removes parameters that contribute little to a trained neural network's output, from individual weights up to whole filters, attention heads, or layers, to make the model smaller and cheaper to run. Most networks are over-parameterized, so a large share of their weights can go with little loss of accuracy, especially when the model is fine-tuned afterward. A 2015 prune-and-retrain method cut AlexNet from 61 million to 6.7 million parameters and VGG-16 from 138 million to 10.3 million with no loss of ImageNet accuracy. Pruning is a standard step in preparing models for phones, cameras, and other edge AI hardware, and it is now also applied to large language models (LLMs).

How Model Pruning Works#

Pruning scores each candidate weight or structure by its importance, removes the lowest-scoring fraction, and fine-tunes the remaining model weights to recover accuracy. The simplest and still most common score is magnitude: weights near zero, or filters with a small L1 or L2 norm, change the output least when removed. Other criteria use gradients or activations to estimate how much the loss would rise without each part. Removing a large fraction at once tends to hurt accuracy, so pruning usually runs in rounds, each followed by fine-tuning, until the model reaches its size target or accuracy drops below tolerance:

Pruning also changed how researchers think about network size. The lottery ticket hypothesis proposes that a dense network contains small subnetworks which, trained in isolation from their original initial weights, reach the full network's accuracy, and its authors consistently found such subnetworks under 10-20% of the original size on MNIST and CIFAR-10.

Unstructured vs. Structured Pruning#

Unstructured pruning zeroes individual weights anywhere in a tensor. It reaches high sparsity with little accuracy loss, but the tensors keep their shape, so standard dense kernels do the same work and speedups need sparse kernels or hardware support. NVIDIA GPUs since the Ampere generation run one such pattern, 2:4 sparsity with two zeros in every four consecutive weights, at up to twice the dense math rate. Structured pruning removes whole filters, channels, heads, or layers, leaving a smaller dense model that runs faster on any processor, at a higher accuracy cost for the same number of removed weights. Removing a filter also removes the matching input channel of the next layer, and libraries such as Torch-Pruning track these dependencies automatically. PyTorch's pruning utilities show the difference between the two styles:

import torch
from torch.nn.utils import prune

# Two identical convolution layers, one for each pruning style
torch.manual_seed(0)
a = torch.nn.Conv2d(16, 32, kernel_size=3)
b = torch.nn.Conv2d(16, 32, kernel_size=3)

# Unstructured: zero the 50% of individual weights with the smallest magnitude
prune.l1_unstructured(a, name="weight", amount=0.5)

# Structured: zero the 50% of output filters with the smallest L2 norm
prune.ln_structured(b, name="weight", amount=0.5, n=2, dim=0)

for name, layer in [("unstructured", a), ("structured", b)]:
    zeros = (layer.weight == 0).float().mean().item()
    empty = (layer.weight.flatten(1) == 0).all(dim=1).sum().item()
    print(f"{name}: {zeros:.0%} of weights zero, {empty} of 32 filters empty")

Both layers end up with 50% of their weights at zero. Only the structured layer has empty filters, 16 of 32, and deleting them leaves a dense layer with half the filters, which halves its computation on ordinary hardware. Hardware built for sparsity benefits from the unstructured kind as well: Ambarella's CVflow integration for Ultralytics YOLO, maintained by Ambarella in its own fork of the package, prunes weights progressively during YOLO training and keeps that sparsity through ONNX export for compilation to its edge AI chips.

Pruning vs. Related Techniques#

• Model quantization: Lowers the precision of every weight, for example from 32-bit floats to 8-bit integers, and keeps all of them. Pruning and quantization are often combined, since pruning reduces the number of operations and quantization makes each one cheaper.
• Knowledge distillation: Trains a smaller student model to imitate a larger teacher, and it often follows structured pruning to restore the accuracy that removing whole filters or layers costs.
• Dropout: Zeroes random activations temporarily during training as a form of regularization, and the full network is used at inference.

Pruning Large Language Models#

Retraining an LLM after every pruning round costs too much, so one-shot methods prune it in a single pass using a small calibration set. SparseGPT pruned the 175-billion-parameter OPT and 176-billion-parameter BLOOM models to 60% unstructured sparsity in under 4.5 hours with a negligible increase in perplexity, and it also supports the 2:4 pattern GPUs accelerate. Structured pruning combined with knowledge distillation produces smaller dense models: NVIDIA's Minitron derived 8B and 4B models from a pretrained 15B model by pruning embedding channels, MLP neurons, and attention heads, then retrained them with up to 40 times fewer tokens than training from scratch.