Hard Negative Mining

Hard negative mining is a training strategy that finds the negative examples a model gets most wrong and concentrates learning on them. A negative example does not belong to the target class, and a hard negative is one the model still scores like a positive: a background patch a detector boxes as an object, or an unrelated image that an embedding model places next to the query. Most negatives are easy, and once a model handles them they contribute almost no learning signal, so selecting the hard ones sharpens decision boundaries and cuts false positives.

How Hard Negative Mining Works#

What counts as hard depends on the task:

• In object detection, a background region is hard when the detector scores it as an object.
• In classification, a hard negative is a wrong-class example assigned high confidence for the target class.
• In retrieval and recommendation systems, it is an irrelevant item ranked close to the query, and high-scoring negatives have the most influence on the gradient.
• In metric learning, it is a negative that sits closer to an anchor than the matching positive does, which a triplet margin loss penalizes.

Mining happens either offline or online. Offline mining runs a trained model over a large pool of negative-only data, collects its false positives, adds them to the training set, and retrains. Classic sliding-window pedestrian detectors were trained this way, scanning person-free images for false alarms and retraining on them. Online mining selects hard examples inside every training batch. SSD ranks its negative default boxes by classification loss and keeps only the highest-loss ones, at most 3 negatives per positive, and online hard example mining (OHEM) applies the same idea to the region proposals of two-stage detectors. The SSD selection takes a few lines:

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Simulated objectness logits for 8,400 candidate boxes: 20 cover an object, the rest are background
logits = torch.randn(8400) - 3
labels = torch.zeros(8400)
labels[:20] = 1
loss = F.binary_cross_entropy_with_logits(logits, labels, reduction="none")

# Keep every positive plus only the highest-loss negatives, at most 3 per positive
positives = labels == 1
negative_loss = loss.masked_fill(positives, 0)
hard_negatives = negative_loss.topk(3 * int(positives.sum())).indices
kept = torch.cat([positives.nonzero().squeeze(1), hard_negatives])

print(f"boxes in the loss: {len(kept)} of {len(loss)}")
print(f"mean negative loss: all {loss[~positives].mean():.3f}, mined {loss[hard_negatives].mean():.3f}")

The loss keeps 80 of the 8,400 boxes: the 20 positives and the 60 negatives the model scores most like objects. Those 60 mined negatives average a loss of 0.648, against 0.075 across all 8,380 negatives, so the gradient concentrates on the background the model actually confuses.

Hard Negative Mining vs. Related Techniques#

• Negative sampling: Picks a manageable subset of all negatives, often at random. Hard negative mining picks the most confusing ones.
• Focal loss: Keeps every example in the loss but down-weights the well-classified ones, which let one-stage detectors such as RetinaNet train on all candidate boxes without explicit sampling.
• Active learning: Selects uncertain unlabeled examples for human annotation, including possible positives. Hard negative mining targets examples already confirmed as negative.
• Class rebalancing: Corrects unequal class frequencies by downsampling and upweighting. A frequent negative class is not necessarily a hard one.
• Data augmentation: Creates variations of existing samples, and cannot introduce a confusing object the dataset never contained.

Real-World Applications#

• Warehouse safety detection: A forklift detector that repeatedly flags pallet jacks, shelving reflections, and scissor lifts erodes operator trust with false alerts. Collecting those scenes as negative images and retraining teaches the detector the features that separate a forklift from similar equipment.
• Visual product search: When a search for a specific black running shoe ranks similar shoes from other product lines too highly, those products become hard negatives. Training on them, for example as the negatives in a Siamese network's anchor-positive-negative triplets, teaches the embedding model to keep subtle differences such as sole shape, logo placement, and material.

Offline Mining Workflow#

Each round of offline mining runs the current model, confirms what it flagged, and folds the confirmed cases back into training:

The review step matters because a "false positive" is often a real object that annotators missed, and adding it as a negative would teach the model to suppress true detections. Ultralytics YOLO treats a training image with a missing or empty label file as a background image, so every anchor point on it receives a zero target score and any confident prediction there is penalized, which makes adding mined negatives to a YOLO26 dataset a matter of copying the images in without labels. Progress shows up as higher precision at the same recall, tracked on the full test set and on a dedicated slice of hard negatives.

Training on only the very hardest negatives can backfire. In FaceNet face-embedding training, picking the hardest negatives early could collapse the model into mapping every face to the same point, so its authors trained on semi-hard negatives, which lie farther from the anchor than the positive but still inside the triplet margin.