Convolution

Convolution is the operation of sliding a small grid of weights, called a kernel or filter, across an input and computing a weighted sum at every position. In computer vision (CV), the input is an image or a stack of feature maps, and each kernel produces an output map that is high wherever its pattern appears: a vertical edge, a patch of texture, or, deeper in a network, part of a wheel or a face. Hand-designed kernels have long been used in image processing for blurring, sharpening, and edge detection. In a convolutional neural network (CNN), the kernel values are model weights learned during training, so the network discovers its own feature detectors.

How Convolution Works#

At each position, the kernel is laid over a patch of the input, each kernel weight is multiplied by the input value beneath it, and the products are summed, plus a bias, into one output value. Moving the kernel across the full height and width produces a two-dimensional output map. Real inputs have several channels, such as the red, green, and blue channels of an image, so each kernel spans all input channels: a 3×3 kernel on a 64-channel input holds 3 × 3 × 64 = 576 weights and still produces one output channel. A convolution layer applies many such kernels, and the number of kernels sets the number of output channels, so a layer from 64 to 128 channels with 3×3 kernels has 3 × 3 × 64 × 128 = 73,728 weights.

The same kernel is reused at every position, which is why convolution layers need far fewer parameters than fully connected layers and why a pattern learned in one corner of an image is recognized everywhere else. Shifting the input shifts the output by the same amount, a property called translation equivariance. Each output value depends only on a small neighborhood of the input, and stacking layers widens the receptive field that a deep output value summarizes, from a few pixels to most of the image.

Deep learning libraries actually compute cross-correlation: mathematical convolution flips the kernel before sliding it, and PyTorch's Conv2d does not. Because the kernel is learned, the network simply learns the flipped weights, and the two are interchangeable in practice. GPU libraries such as cuDNN usually recast convolution as a large matrix multiplication or use Winograd or FFT-based algorithms that compute the same result faster than sliding the kernel literally.

Stride, Padding, Dilation, and Groups#

Four hyperparameters control the shape of a convolution's output and the cost of computing it:

• Stride: The step between kernel positions. A stride of 2 skips every other position and halves the output's height and width, which is how many modern CNNs downsample.
• Padding: Extra values, usually zeros, added around the input border. Padding of 1 keeps a 3×3 convolution's output the same size as its input at stride 1.
• Dilation: Spaces the kernel's taps apart, so a 3×3 kernel with dilation 2 covers a 5×5 area with the same nine weights, enlarging the receptive field at no extra cost.
• Groups: Splits the channels into independent groups. With one group per channel, a depthwise convolution filters each channel separately, and a following 1×1 pointwise convolution mixes the channels, a pair called a depthwise separable convolution that MobileNets popularized for mobile hardware.

For an input of size n, kernel size k, padding p, stride s, and dilation d, the output size is ⌊(n + 2p − d(k − 1) − 1) / s⌋ + 1. The snippet below applies four layers that each map a 64-channel, 80×80 feature map to 128 channels:

import torch
from torch import nn

x = torch.randn(1, 64, 80, 80)  # a 64-channel, 80x80 feature map

# Four ways to map 64 input channels to 128 output channels
layers = {
    "standard 3x3": nn.Conv2d(64, 128, kernel_size=3, padding=1, bias=False),
    "stride 2": nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1, bias=False),
    "dilation 2": nn.Conv2d(64, 128, kernel_size=3, padding=2, dilation=2, bias=False),
    "depthwise separable": nn.Sequential(
        nn.Conv2d(64, 64, kernel_size=3, padding=1, groups=64, bias=False),  # one 3x3 filter per channel
        nn.Conv2d(64, 128, kernel_size=1, bias=False),  # 1x1 convolution mixes channels
    ),
}
for name, layer in layers.items():
    params = sum(p.numel() for p in layer.parameters())
    print(f"{name}: output {tuple(layer(x).shape)}, {params:,} weights")

The standard layer outputs (1, 128, 80, 80) with 73,728 weights. Stride 2 halves the output to (1, 128, 40, 40) with the same weights, and dilation 2 keeps 80×80 and 73,728 weights while each output sees a 5×5 area. The depthwise separable pair produces the same (1, 128, 80, 80) output from only 8,768 weights, about 8.4 times fewer. YOLO26 uses both ideas: five stride-2 3×3 convolutions shrink a 640×640 image to the 20×20 grid of its deepest backbone feature map, and the classification branch of its detection head pairs depthwise 3×3 and pointwise 1×1 convolutions to cut computation.

Convolution in Other Dimensions#

• 1D convolution: Slides along a single axis, for audio waveforms, sensor signals, and time series.
• 3D convolution: Adds a depth or time axis, for video clips and volumetric scans such as CT and MRI in medical image analysis.
• Transposed convolution: Spreads each input value across a larger output, upsampling feature maps in segmentation models such as U-Net. It restores the spatial size that a strided convolution reduced, using its own learned weights.

Convolution vs. Related Operations#

• Fully connected layer: Connects every input to every output with its own weight. A convolution is a fully connected layer restricted to local neighborhoods with weights shared across positions.
• Pooling: Slides a window like convolution but applies a fixed rule, such as taking the maximum, with no learned weights.
• Self-attention: Computes data-dependent weights between every pair of positions, so it relates distant parts of an image in a single layer. A convolution applies the same fixed kernel everywhere and sees only a local window, which keeps its cost linear in the number of pixels.
• Image filtering: Classic filters such as Gaussian blur and the Sobel edge detector are convolutions with fixed, hand-designed kernels. OpenCV's filter2D also computes correlation, matching deep learning libraries, and flipping the kernel turns it into true convolution.