Anchor-Based Detectors

Anchor-based detectors are object detection models that tile predefined anchor boxes over an image and, for each anchor, predict class scores and the offsets that turn it into a fitted bounding box. The family includes two-stage models such as Faster R-CNN, whose region proposal network scores anchors to produce candidate regions, and one-stage models such as SSD, RetinaNet, and YOLOv2 through YOLOv5, which classify and refine every anchor directly. Anchors were the standard way to localize objects from Faster R-CNN in 2015 until anchor-free detectors such as FCOS matched their accuracy in 2019.

How Anchor-Based Detection Works#

A backbone and neck turn the image into feature maps, and a fixed set of anchors, several sizes and aspect ratios, sits at every cell of each map. For every anchor, the detection head predicts one score per class, plus an objectness score in YOLOv2 through YOLOv5, and four regression values. The widely used encoding from Faster R-CNN shifts the anchor's center by fractions of its width and height and scales its width and height exponentially:

import torch

# One anchor as (center x, center y, width, height) in pixels
anchor = torch.tensor([320.0, 240.0, 64.0, 128.0])

# Offsets (tx, ty, tw, th) that the regression branch predicts for this anchor
tx, ty, tw, th = torch.tensor([0.25, -0.10, 0.20, -0.05])

# Faster R-CNN decoding: shift the center in units of anchor size, scale width and height exponentially
cx = anchor[0] + tx * anchor[2]
cy = anchor[1] + ty * anchor[3]
w = anchor[2] * torch.exp(tw)
h = anchor[3] * torch.exp(th)
print(f"anchor 64x128 at (320, 240) -> box {w:.1f}x{h:.1f} at ({cx:.1f}, {cy:.1f})")

The anchor becomes a 78.2 × 121.8 pixel box centered at (336.0, 227.2): it moved 16 pixels right and 12.8 pixels up, grew 22% wider (exp(0.2)), and shrank 5% in height (exp(−0.05)). Because the offsets are relative to the anchor's size, the same predicted values work for small and large anchors.

At inference, the detector decodes every anchor this way, drops boxes below a confidence threshold, and runs non-maximum suppression (NMS) to remove duplicates from neighboring anchors.

Training: Labeling Each Anchor#

Training labels every anchor by its Intersection over Union (IoU) with the labeled boxes. RetinaNet uses these thresholds:

Faster R-CNN's region proposal network uses stricter values, 0.7 and 0.3. RetinaNet's 9 anchors per location on five pyramid levels add up to about 100,000 anchors per image, nearly all of them negatives, and its focal loss down-weights those easy negatives so they do not overwhelm training.

Major Anchor-Based Architectures#

• Two-stage detectors: Faster R-CNN's region proposal network scores anchors as object or background and refines them into about 300 proposals per image, and a second head classifies and refines each proposal. With a VGG-16 backbone it ran at 5 FPS on a GPU; Mask R-CNN adds a mask branch to the same design.
• One-stage object detectors: SSD, RetinaNet, and the YOLO models from YOLOv2 to YOLOv5 classify and refine every anchor in one pass. YOLOv2 replaced hand-picked anchors with k-means clusters of the training boxes' widths and heights.

Ultralytics YOLOv5 is anchor-based, with 3 anchors on each of its three output levels, and it matches labels to anchors by shape: an anchor is a candidate when a label's width and height are both within 4× of the anchor's. The Ultralytics package's YOLOv5u variants replace that head with the anchor-free split head introduced in YOLOv8, the design YOLO26 continues.

Anchor-Based vs. Anchor-Free Detectors#

Anchor-free detectors predict a box from each grid point, as distances to the four edges, or from keypoints such as object centers, so they have no anchor sizes, aspect ratios, or IoU thresholds to tune and produce one candidate per location instead of one per anchor shape. ATSS traced the accuracy gap between RetinaNet and the anchor-free FCOS to how each chose positive training samples: with the same selection rule, the two reached the same accuracy, and RetinaNet scored 39.2 AP on COCO minival with 9 anchors per location and 39.3 AP with 1.