Anchor-Free Detectors

Anchor-free detectors are object detection models that predict each bounding box directly from a point on a feature map, or from keypoints such as an object's corners or center, without a set of predefined anchor boxes. A location near an object predicts the distances from itself to the object's four edges, or a heatmap peak marks an object's center and the network regresses its width and height. This removes the anchor sizes, aspect ratios, and anchor-matching thresholds that anchor-based detectors must tune for each dataset, and anchor-free heads have become the standard design for real-time detection.

How Anchor-Free Detection Works#

Anchor-free detectors fall into two families:

• Point-based detectors such as FCOS: Every cell of every feature map level is a candidate. FCOS (2019) predicts class scores and four distances, to the left, top, right, and bottom edges, at each location inside an object, plus a centerness score that down-weights boxes predicted from points far from the object's center.
• Keypoint-based detectors such as CornerNet and CenterNet: CornerNet (2018) predicts heatmaps of top-left and bottom-right corners and pairs the corners that belong to the same object with learned embeddings. CenterNet predicts a heatmap of object centers and regresses width and height at each peak, extracting peaks with a 3 × 3 max-pooling operation so it needs no IoU-based non-maximum suppression (NMS).

Anchor-free prediction also predates anchors in the YOLO line: YOLOv1 in 2015 predicted boxes directly from the grid cell containing each object's center, and adding anchor boxes in YOLOv2 raised recall from 81% to 88% at a 0.3-point mAP cost. YOLOX brought the dense point-based design back to YOLO in 2021, and Ultralytics switched from YOLOv5's anchors to an anchor-free head in YOLOv8 in 2023, so custom datasets no longer need an anchor-fitting step; YOLO26 keeps that design.

Decoding a dense anchor-free prediction needs only the cell's position and the four predicted distances:

import torch

# Center of the grid cell at column 10, row 6 of a stride-16 feature map, in image pixels
stride = 16
point = (torch.tensor([10.0, 6.0]) + 0.5) * stride

# Distances the head predicts from that point to the left, top, right, and bottom box edges, in strides
ltrb = torch.tensor([2.0, 3.5, 1.5, 4.0]) * stride

# Anchor-free decoding: no template box, only the point and four distances
x1, y1 = point - ltrb[:2]
x2, y2 = point + ltrb[2:]
print(f"point {point.tolist()} -> box ({x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f})")

The point at (168, 104) becomes the box (136, 48, 192, 168), 56 pixels wide and 120 pixels tall. A dense head applies this decoding at every cell of every pyramid level, so each cell yields one candidate box.

Choosing Which Points Learn Each Object#

With no anchors to compare against labeled boxes by Intersection over Union (IoU), an anchor-free detector needs another rule for which points count as positive training samples. FCOS uses position and size: a point is positive if it lies inside a labeled box and its largest distance to that box's edges fits the range of the point's pyramid level, 0 to 64 pixels on the stride-8 level, 64 to 128 on stride 16, and so on up to stride 128.

Later detectors choose positives adaptively. ATSS showed that the accuracy gap between the anchor-based RetinaNet and the anchor-free FCOS came mainly from how each defined positive and negative samples, and that with the same sampling rule, tiling several anchors per location brought no gain. Task-aligned assignment, introduced by TOOD and adopted by PP-YOLOE and many later real-time detectors, ranks the points inside each object by a weighted product of predicted class score and box IoU and keeps the top few as positives.

Anchor-Free vs. Anchor-Based Detectors#

• Fewer hyperparameters: No anchor sizes, aspect ratios, or IoU matching thresholds have to be fitted to each dataset, a step anchor-based detectors often handle with k-means clustering of the training boxes.
• Shape flexibility: Point-based assignment depends only on where an object lies, so an object whose shape resembles no anchor, such as a long thin pipe, still gets positive samples.
• Shared vocabulary: Many anchor-free codebases still call their grid points "anchor points," meaning a location with no size or aspect ratio.
• Fewer predictions: Each location predicts one box instead of one per anchor shape, so decoding and NMS handle several times fewer candidates; RetinaNet's 9 anchors per location add up to about 100,000 candidates per image.