Non-Maximum Suppression (NMS)

Non-maximum suppression (NMS) is the post-processing step that turns a detector's raw, overlapping predictions into one box per object. A dense object detection model scores thousands of candidate bounding boxes per image, and several neighboring candidates usually fire on the same object with slightly different positions and scores. NMS keeps the highest-confidence box in each overlapping cluster and suppresses the rest, using intersection over union (IoU) to decide which boxes overlap enough to count as duplicates.

How Non-Maximum Suppression Works#

Greedy NMS, the standard version, first drops candidates below a confidence threshold and sorts the rest by score. It then keeps the top box, removes every remaining box whose IoU with it exceeds the IoU threshold, and repeats with the best box still standing until none are left. Detectors normally run it once per class, so a person box never suppresses an overlapping bicycle box.

torchvision's nms implements greedy NMS for one class and returns the indices of the boxes it keeps, sorted by score.

import torch
from torchvision.ops import nms

# Candidate boxes as [x1, y1, x2, y2] with one confidence score each
boxes = torch.tensor(
    [
        [100, 100, 200, 200],  # A
        [105, 105, 195, 195],  # B: near-duplicate of A (IoU 0.81)
        [300, 300, 400, 400],  # C: a separate object
    ],
    dtype=torch.float32,
)
scores = torch.tensor([0.90, 0.80, 0.95])

# Suppress any box whose IoU with a higher-scoring kept box exceeds 0.5
keep = nms(boxes, scores, iou_threshold=0.5)
print(keep.tolist())

The output is [2, 0]: box C is kept first because it has the highest score, box A is kept because it does not overlap C, and box B is removed because its IoU with A is 0.81.

Tuning the Thresholds#

The confidence threshold removes weak candidates before suppression starts. The IoU threshold sets how much overlap counts as a duplicate. When it is too low, two real objects standing close together, such as people in a crowd, merge into one detection and recall drops. When it is too high, duplicate boxes survive and precision drops. Class-agnostic NMS lets any box suppress any other regardless of class, which removes double labels when a model gives one object two similar classes, such as "car" and "truck."

Limitations of NMS#

• Crowded scenes: a single fixed IoU threshold cannot separate a duplicate box from a second object that genuinely overlaps the first, so heavily occluded objects get suppressed.
• Fixed heuristic: the threshold is chosen by hand, sits outside the trained network, and is not learned from data.
• Latency and deployment: the greedy loop runs after the network and its cost grows with the number of candidates, which adds inference latency. Export formats that cannot embed NMS in the model graph output raw candidates, and the application must run NMS itself.

NMS Variants and NMS-Free Detection#

Soft-NMS lowers the scores of overlapping boxes as a function of their overlap instead of deleting them, which recovers more objects in crowds. DIoU-NMS, proposed with the Distance-IoU loss, also considers the distance between box centers, so two boxes whose centers are far apart are less likely to be suppressed.

End-to-end detectors remove the step altogether. DETR is trained with one-to-one Hungarian matching between predictions and labels, so each object receives exactly one prediction, and YOLOv10 brought a one-to-one head to real-time YOLO models. YOLO26 trains both kinds of head and applies NMS by default, because its one-to-many head scores slightly higher, 40.9 versus 40.1 mAP50-95 on COCO for YOLO26n. Passing nms=False selects the one-to-one head, which outputs at most 300 final boxes per image with no NMS pass.