Identity Mapping
Identity mapping passes a block's input unchanged through a shortcut and adds it to the output, so gradients reach the early layers of deep networks.
In deep learning, an identity mapping is a connection that carries a block's input forward unchanged and adds it to the block's output, so the block computes y = x + F(x) instead of y = F(x). The name comes from the identity function, f(x) = x. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun built residual networks (ResNet) on these identity shortcuts in 2015 and won that year's ImageNet classification challenge with a 152-layer network, and identity shortcuts now run through nearly every deep architecture, from convolutional neural networks (CNNs) to the residual stream of every transformer.
Why Identity Mappings Make Depth Trainable#
Before ResNet, adding layers often made networks worse. He and colleagues found that a 56-layer plain network had higher training error on CIFAR-10 than a 20-layer one, so the extra depth hurt optimization itself, a problem they called degradation. A deeper network can always match a shallower one by setting its extra layers to the identity, yet stacks of nonlinear layers struggle to learn the identity. A residual block only has to learn the difference F(x) = H(x) - x from its input, and driving F toward zero recovers the identity.
The shortcut also changes backpropagation. Because y = x + F(x), the derivative of y with respect to x is 1 plus the derivative of F, and that constant 1 passes the gradient straight back through every block. In a plain stack, the gradient is a product of many per-layer factors, and when those factors are below 1 the result shrinks exponentially, the vanishing gradient problem. Running the same 50 layers with and without shortcuts in PyTorch shows the difference:
import torch
from torch import nn
torch.manual_seed(0)
# 50 identical-shape layers, reused for a plain and a residual stack
layers = [nn.Sequential(nn.Linear(64, 64), nn.Tanh()) for _ in range(50)]
x = torch.randn(32, 64, requires_grad=True)
for residual in (False, True):
h = x
for layer in layers:
h = h + layer(h) if residual else layer(h) # identity shortcut adds the input back
loss = h.square().mean()
(grad,) = torch.autograd.grad(loss, x)
print(f"residual={residual}: gradient norm at the input {grad.norm():.1e}")It prints a gradient norm of 1.1e-15 at the input for the plain stack and 2.8e+00 for the residual stack built from the same weights. Without shortcuts, the first layers of this network would receive almost no training signal.
Keeping the Shortcut Clean#
In their 2016 follow-up, Identity Mappings in Deep Residual Networks, the same authors tested shortcuts that scale, gate, convolve, or apply dropout to x, and every variant trained worse than a pure identity. They also moved batch normalization and the activation function inside the residual branch, before the weights, so nothing touches the signal between blocks. This pre-activation design trained a 1,001-layer ResNet on CIFAR-10 to 4.62% error and a 200-layer ResNet on ImageNet. Modern transformers follow the same rule with pre-norm blocks, which apply normalization inside each attention and feed-forward branch and leave the residual stream itself untouched.
An identity shortcut needs the input and output to have the same shape. Where a block changes the number of channels or downsamples, ResNet substitutes a projection shortcut, usually a 1 x 1 convolution with matching stride:
Ultralytics YOLO26 uses identity shortcuts throughout its backbone and neck: every C3k2 block is built from bottlenecks that return x plus the output of two 3 x 3 convolutions whenever their input and output channels match, and its SPPF block adds its input back to its pooled output the same way.
Identity Initialization#
A block can also start as an identity. Setting the scale of the last batch normalization layer in each residual branch to zero makes F(x) = 0 at initialization, so a deep network begins training as if it were shallow; Priya Goyal and colleagues used this to train ResNet-50 on ImageNet in one hour with large batches, and torchvision exposes it as zero_init_residual=True. ReZero, proposed in 2020, multiplies each residual branch by a learnable scalar that starts at zero.
PyTorch's nn.Identity module is a literal identity map with no parameters. Replacing a classifier head with it, as in model.fc = nn.Identity(), turns an image classification network into a feature extraction model that returns its penultimate embeddings.
Related Terms#
Skip connection is the broader term for any path that bypasses layers. A residual connection is the additive kind, and an identity mapping is a residual connection whose path applies no transformation. U-Net and DenseNet use skip connections that concatenate earlier feature maps instead of adding them. Identity preservation is an unrelated term from generative AI and tracking, meaning that a generated or tracked person keeps the same recognizable appearance across images or video frames.










