Anchor Boxes

Anchor boxes are predefined rectangles of fixed sizes and aspect ratios, tiled at every position of a detector's feature maps, that serve as starting guesses for where objects are and what shape they have. An anchor-based detector scores each anchor for every class and predicts small offsets that move and resize it to fit the object, which is easier to learn than predicting a box from nothing. Faster R-CNN places 9 anchors at each position, three areas (128², 256², and 512² pixels) times three aspect ratios (1:1, 1:2, and 2:1), which comes to roughly 20,000 anchors for a 1000 × 600 image.

How Anchors Are Defined#

An anchor set is a list of sizes and aspect ratios repeated at every cell of a feature map, so the number of anchors equals cells × shapes per cell. Detectors with a feature pyramid network (FPN) assign small anchors to fine, high-resolution levels and large anchors to coarse levels: RetinaNet uses 9 anchors per location on each of five levels, about 100,000 per image. SSD calls them default boxes, and SSD300 tiles 8,732 of them.

Hand-picked shapes fit some datasets poorly, so YOLOv2 ran k-means clustering on the widths and heights of the training boxes, using 1 − IoU as the distance, and kept 5 cluster centers as its anchors. YOLOv3 clustered 9 anchors and split them evenly across its three output strides, from 10 × 13 pixels on the stride-8 level to 373 × 326 on the stride-32 level.

Matching Anchors to Objects#

During training, each anchor is compared with every labeled box by Intersection over Union (IoU). Anchors that overlap an object enough become positive samples and learn that object's class and the offsets to its bounding box, anchors with little overlap become negatives that learn "background," and anchors in between are ignored. Faster R-CNN's region proposal network marks IoU of 0.7 or more as positive and below 0.3 as negative, and it also makes the best-matching anchor of each object positive, so no object is left without one.

import torch
from torchvision.ops import box_iou

# Nine anchors centered on one feature-map cell at (200, 200): 3 sizes x 3 aspect ratios (height / width)
shapes = [(size, ratio) for size in (64, 128, 256) for ratio in (0.5, 1.0, 2.0)]
anchors = []
for size, ratio in shapes:
    w, h = size / ratio**0.5, size * ratio**0.5
    anchors.append([200 - w / 2, 200 - h / 2, 200 + w / 2, 200 + h / 2])

# A labeled pedestrian box (x1, y1, x2, y2): tall and narrow
person = torch.tensor([[160.0, 110.0, 240.0, 290.0]])

# Faster R-CNN RPN rule: IoU >= 0.7 is positive, below 0.3 is negative, anything between is ignored
iou = box_iou(torch.tensor(anchors), person).squeeze(1)
for (size, ratio), value in zip(shapes, iou.tolist()):
    label = "positive" if value >= 0.7 else "negative" if value < 0.3 else "ignored"
    print(f"size {size}, ratio {ratio}: IoU {value:.2f} -> {label}")

Only the tall 128-pixel anchor (ratio 2.0) overlaps the pedestrian enough to be positive, at IoU 0.88. Two other 128-pixel anchors, at IoU 0.31 and 0.50, are ignored, and the six anchors of the wrong size are negatives. The positive anchor then learns offsets in the standard parameterization: the center shifts by fractions of the anchor's width and height, and the width and height scale by exponentials of the predicted values.

Choosing Anchor Sizes for a Dataset#

Anchors that match a dataset's object shapes give more objects a positive anchor, which raises the recall a detector can reach. Ultralytics YOLOv5 starts from those 9 YOLOv3 anchors and checks them before training with AutoAnchor: it measures the best possible recall of its anchors on the training labels and, unless it exceeds 0.98, re-fits them with k-means and 1,000 generations of genetic evolution, keeping the new anchors only if recall improves.

Limitations and Anchor-Free Detectors#

Anchors bring three costs. Nearly all of the tens of thousands of anchors in an image cover background, which swamps the loss with easy negatives, and RetinaNet introduced focal loss to down-weight them. Several neighboring anchors fit the same object, so the output needs non-maximum suppression (NMS) to keep one box per object. Sizes, aspect ratios, and IoU thresholds are also hyperparameters that change with every dataset and image size.

Anchor-free detectors remove those hyperparameters by predicting boxes from points or keypoints, and most real-time detectors released since YOLOX in 2021 are anchor-free. Anchors remain in anchor-based detectors such as Faster R-CNN, Mask R-CNN, and RetinaNet. Comparing RetinaNet with the anchor-free FCOS, ATSS found that once both chose positive samples the same way, RetinaNet with 9 anchors per location was no more accurate than with 1.