Two-Stage Object Detectors

Two-stage object detectors are object detection models that split the job into two steps: a first stage proposes regions of the image that might contain an object, and a second stage classifies each region and refines its bounding box. The R-CNN family defined the design between 2014 and 2018 and led detection and instance segmentation benchmarks for years. One-stage object detectors skip the proposal step and predict every box from a dense grid in one pass, which is why most real-time systems use them.

How Two-Stage Detectors Work#

A backbone convolutional neural network computes feature maps for the whole image once, usually with a feature pyramid network (FPN) on top. The first stage, a region proposal network (RPN), scores a set of anchor boxes at every feature-map position for objectness and regresses offsets that reshape them; Faster R-CNN uses nine anchors per position, three sizes times three aspect ratios. Non-maximum suppression (NMS) removes overlapping proposals, and the top-scoring regions, 300 per image at test time in the original paper, go to the second stage. There, RoI pooling or its more precise successor, RoIAlign, crops a fixed-size feature grid for each proposal, and a small head predicts a class, including a background class, and class-specific box offsets for every crop.

A forward hook on the RPN of torchvision's Faster R-CNN shows how much work the second stage receives:

import urllib.request

from PIL import Image
from torchvision.models.detection import FasterRCNN_ResNet50_FPN_V2_Weights, fasterrcnn_resnet50_fpn_v2

# Load Faster R-CNN with a ResNet-50 FPN backbone, pretrained on COCO
weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn_v2(weights=weights).eval()
image = Image.open(urllib.request.urlopen("https://ultralytics.com/images/bus.jpg"))

# Report how many regions the first stage, the region proposal network, passes on
model.rpn.register_forward_hook(lambda module, inputs, output: print(f"Stage 1: {len(output[0][0])} proposals"))

# The second stage classifies and refines every proposal
detections = model([weights.transforms()(image)])[0]
labels = detections["labels"][detections["scores"] > 0.5]
print(f"Stage 2: {len(labels)} objects {[weights.meta['categories'][i] for i in labels]}")

It prints Stage 1: 1000 proposals and Stage 2: 6 objects ['person', 'bus', 'person', 'person', 'person', 'tie']: torchvision keeps 1,000 proposals per image at inference, and the second stage turns them into six confident detections.

The R-CNN Family#

• R-CNN (2014): Selective search proposed about 2,000 regions, each region was warped to a fixed size and passed through the CNN separately, and support vector machines classified the features, so detection with a VGG-16 backbone took 47 seconds per image on a GPU.
• Fast R-CNN (2015): Computing the feature map once and pooling each proposal's features from it with RoI pooling trained VGG-16 9 times faster than R-CNN and tested 213 times faster.
• Faster R-CNN (2015): The RPN replaced selective search and shares the backbone's features, so proposals cost almost nothing and the full detector ran at 5 frames per second.
• Mask R-CNN (2017): A third branch predicts a pixel mask for every region, turning the detector into an instance segmentation model, and RoIAlign samples features at exact fractional positions where RoI pooling rounded them.
• Cascade R-CNN (2018): Three second-stage heads trained at rising IoU thresholds of 0.5, 0.6, and 0.7 each refine the boxes of the one before, tightening localization.

Two-Stage vs. One-Stage Detectors#

The RPN discards most of the background before the classifier sees it, so the second stage trains on sampled regions in which objects make up a fixed share, 25% in Fast R-CNN, and classifies each region with features pooled at its exact location. One-stage detectors score every position of a dense grid, about 100,000 candidates per image in RetinaNet, and trailed two-stage accuracy until focal loss down-weighted the easy background candidates in 2017. Current one-stage models match or exceed two-stage accuracy at a fraction of the latency, and a single network is simpler to export to edge hardware, so two-stage models now appear mostly as research baselines and in libraries such as Detectron2. Ultralytics' auto_annotate applies the same propose-then-refine split across two models for data annotation: a YOLO26 detector proposes boxes, and Meta's Segment Anything Model (SAM), prompted with each box, segments the object inside it to write segmentation labels.

Real-World Applications#

• Medical imaging: Mask R-CNN variants detect and outline individual cells, nuclei, and lesions in microscopy and CT scans, where every instance needs its own mask for counting and measurement.
• Document layout analysis: The PubLayNet benchmark of more than 360,000 page images trained Faster R-CNN and Mask R-CNN models to locate text blocks, titles, lists, tables, and figures before text recognition.
• Aerial imagery: Oriented R-CNN extends the RPN to propose rotated regions, so the second stage outputs oriented bounding boxes for ships, planes, and vehicles in the DOTA dataset.