Feature Pyramid Network (FPN)

A Feature Pyramid Network (FPN) is a neck architecture that turns a backbone's multi-resolution feature maps into a pyramid in which every level carries strong semantic information and is matched to a range of object sizes. Introduced by Facebook AI Research in Feature Pyramid Networks for Object Detection (CVPR 2017), it adds a top-down pathway and lateral connections to a standard convolutional neural network (CNN), so small objects can be detected on high-resolution levels that still carry the deep layers' knowledge of what objects look like. FPNs and their descendants sit between the backbone and the detection head in most modern object detection architectures and instance segmentation models.

Why Detectors Need Multi-Scale Features#

A CNN backbone downsamples as it goes deeper, so its last feature map has strong semantics and coarse resolution: at stride 32, a 20-pixel object covers less than one cell of the grid. Its early feature maps keep fine spatial detail and respond mostly to edges and textures. Before FPN, detectors had three options, each with a cost. Running the network on an image pyramid at several scales is accurate and several times slower. Predicting only from the last map, as the original Faster R-CNN did, is fast and weak on small objects. Predicting from several backbone stages, as SSD does, starts high in the network to avoid weak low-level features and so misses the high-resolution maps that small objects need. FPN gets the multi-scale coverage of an image pyramid from a single forward pass, and Faster R-CNN with FPN set a new single-model state of the art on COCO detection while running at 5 frames per second on a GPU.

How an FPN Works#

An FPN builds its pyramid in three steps:

1. Bottom-up pathway: The backbone's normal forward pass. FPN takes the last feature map of each stage, labeled C2 to C5 at strides 4, 8, 16, and 32 in a ResNet.
2. Top-down pathway: Starting from the coarsest level, each map is upsampled by a factor of 2 with nearest-neighbor interpolation, carrying high-level semantics toward the finer levels.
3. Lateral connections: A 1×1 convolution brings each backbone map to a common width of 256 channels, the result is added element-wise to the upsampled map from the level above, and a 3×3 convolution smooths the sum to reduce upsampling artifacts. The outputs, P2 to P5, all have 256 channels, so one prediction head with shared weights runs on every level.

Torchvision ships a ready-made FeaturePyramidNetwork module that takes the backbone maps as an ordered dictionary.

from collections import OrderedDict

import torch
from torchvision.ops import FeaturePyramidNetwork

# Backbone outputs for a 640x640 image at strides 8, 16, and 32 (ResNet-50 channel counts)
features = OrderedDict(
    c3=torch.randn(1, 512, 80, 80),
    c4=torch.randn(1, 1024, 40, 40),
    c5=torch.randn(1, 2048, 20, 20),
)

# Lateral 1x1 convs plus the top-down pathway give every level the same 256 channels
fpn = FeaturePyramidNetwork(in_channels_list=[512, 1024, 2048], out_channels=256)
pyramid = fpn(features)

for name, level in pyramid.items():
    print(name, tuple(level.shape))

It prints c3 (1, 256, 80, 80), c4 (1, 256, 40, 40), and c5 (1, 256, 20, 20): each level keeps its spatial size, and all three now share one channel width, which is what lets a single head serve every scale.

Applications#

• Aerial and satellite imagery: Vehicles, people, and buildings in satellite image analysis and drone footage often span only a few pixels, so detectors rely on the stride-4 and stride-8 levels of the pyramid.
• Instance segmentation: Mask R-CNN, available in Detectron2, assigns each region to a pyramid level by its size before predicting its box and mask.
• Panoptic segmentation: Panoptic FPN adds a semantic segmentation branch to the same pyramid, labeling stuff such as road and sky alongside countable objects.
• Autonomous vehicles: A single camera frame holds nearby trucks that fill half the image and distant pedestrians and traffic lights a few pixels tall.

FPN vs. PANet, BiFPN, and Other Necks#

FPN's top-down pathway was the starting point for a family of necks that change how information flows between levels:

• PANet: Adds a bottom-up path after FPN's top-down path, shortening the route from fine, well-localized features to the top levels. PANet took first place in the COCO 2017 instance segmentation challenge.
• BiFPN: The neck of EfficientDet repeats top-down and bottom-up fusion several times, removes nodes that have only one input, and learns a weight for each input it fuses.
• NAS-FPN: A pyramid layout found with neural architecture search and stacked several times.
• U-Net: Also merges coarse and fine features through skip connections, then predicts once at full resolution, which suits per-pixel segmentation more than multi-scale detection.

YOLO26 uses an FPN-plus-PAN neck that fuses levels by concatenation: two upsampling steps carry stride-32 context down to the stride-8 level, two strided convolutions carry detail back up, and the head predicts from P3, P4, and P5 at strides 8, 16, and 32. Its yolo26-p2.yaml variant extends the pyramid with a stride-4 P2 level for very small objects, which raises YOLO26n's unfused compute from 6.1 to 9.5 GFLOPs.