Object Detection Architectures

Object detection architectures are the network designs that turn an image into a list of bounding boxes, class labels, and confidence scores. Nearly every modern detector is built from a backbone, a neck, and a head, and designs differ mainly in how they generate candidate boxes, from region proposals, a dense grid, or learned queries, and in how they remove duplicate predictions, with non-maximum suppression (NMS) or one-to-one matching. Those choices set a detector's accuracy, its speed, and how easily it deploys to edge hardware. Object detection names the task, and the detection head is one part of an architecture.

Backbone, Neck, and Head#

The backbone extracts features. It is a convolutional neural network such as ResNet or CSPDarknet, or a vision transformer, often pretrained on ImageNet classification, and it downsamples the image in stages so its feature maps lose resolution and gain semantic content with depth. The neck, typically a feature pyramid network (FPN) with an extra bottom-up path as in PANet, merges deep and shallow maps so each output scale carries both fine detail and context. The head predicts boxes and class scores at every location of each scale, usually at strides 8, 16, and 32, so the finest map finds small objects and the coarsest finds large ones. The three parts of a real detector can be read from its model definition:

from ultralytics import YOLO

# Load YOLO26n and split its layers the way its model YAML does
model = YOLO("yolo26n.pt")
layers = model.model.model
n_backbone = len(model.model.yaml["backbone"])

# The backbone extracts features, the neck fuses scales, and the last layer is the head
print("Backbone:", [type(m).__name__ for m in layers[:n_backbone]])
print("Neck:", [type(m).__name__ for m in layers[n_backbone:-1]])
print("Head:", type(layers[-1]).__name__, "at strides", model.model.stride.tolist())

The backbone list shows strided Conv layers and C3k2 blocks ending in SPPF pooling and a C2PSA attention block, the neck runs Upsample, Concat, and C3k2 layers top-down and strided Conv, Concat, and C3k2 layers back up, and a Detect head predicts at strides 8, 16, and 32.

Detector Families#

• Two-stage detectors: A region proposal network suggests a few hundred candidate regions, and a second head classifies and refines each one, as in Faster R-CNN (2015) and Mask R-CNN (2017).
• Anchor-based one-stage detectors: SSD (2016), YOLOv2 to YOLOv5, and RetinaNet (2017) predict offsets from predefined anchor boxes at every grid cell in a single pass.
• Anchor-free detectors: FCOS (2019) and YOLOv8 (2023) predict the distances from a point to the four box edges, CenterNet (2019) predicts box centers as heatmap peaks with a width and height at each, and YOLOX (2021) returned the YOLO family to anchor-free prediction, so none of them needs anchor tuning.
• DETR-style transformer detectors: DETR (2020) decodes a fixed set of learned object queries and matches them one-to-one with the labeled objects, so its output needs no NMS; Deformable DETR reached better accuracy with 10 times fewer training epochs, and RT-DETR made the design real-time in 2023.
• NMS-free CNN detectors: YOLOv10 (2024) trains a one-to-one head alongside a one-to-many head and runs only the one-to-one head at inference, so its boxes need no NMS.
• Open-vocabulary detectors: Grounding DINO and YOLO-World replace the fixed classification layer with text embeddings, so they detect classes named at inference time.

Choosing an Architecture#

Benchmarks such as mean average precision (mAP) on COCO rank accuracy, but the target hardware and the objects in the images usually decide the design. YOLO26 is an anchor-free one-stage CNN with both kinds of head: it applies NMS to its one-to-many branch by default, nms=False switches to its one-to-one branch, and its model YAMLs include P2 and P6 variants that add a stride-4 or stride-64 output.

• Edge and real-time video: Small CNN detectors compile well to runtimes such as TensorRT and OpenVINO, and an NMS-free head removes a postprocessing step whose cost grows with the number of candidates.
• Tiny objects: Aerial and inspection images need a finer output level, such as stride 4, or tiled inference that runs the detector on overlapping crops.
• Changing product catalogs: Open-vocabulary detectors find a new product from a text prompt, with no labeled images of it.
• Rotated objects: Ships, vehicles, and buildings in satellite images fit better in oriented boxes, which add an angle output to the head.