Multi-Object Tracking (MOT)

Multi-object tracking (MOT) is the computer vision task of following every object of interest through a video at once, giving each one a persistent ID so its positions across frames form a trajectory. The number of objects is unknown and changes constantly as people or vehicles enter, leave, cross paths, and disappear behind each other, so a tracker must decide in every frame which detections continue existing tracks, which start new ones, and which tracks have ended. MOT is the most common form of object tracking; single-object tracking (SOT) follows one target chosen in the first frame, regardless of its class.

How Multi-Object Tracking Works#

Most MOT systems use tracking-by-detection: an object detection model finds objects in each frame, and a tracker links those detections to the tracks it already holds. The tracker predicts where each track should appear with a motion model, usually a Kalman filter, builds a cost matrix between predicted tracks and new detections from box overlap (IoU), motion, and optionally appearance embeddings from an object re-identification (Re-ID) model, and solves the matching as a linear assignment problem with the Hungarian algorithm. Matched tracks keep their IDs, confident unmatched detections start new tracks, and tracks without a match are kept as "lost" for a number of frames so they can be recovered after an occlusion.

Crowds make the association step hard, because partly hidden people produce low-confidence detections that a simple threshold throws away. ByteTrack keeps them for a second matching pass:

Tracker Designs#

• SORT and its successors: SORT (2016) combined a Kalman filter, IoU costs, and the Hungarian algorithm. DeepSORT added appearance features to recover identities after occlusion, ByteTrack added the low-confidence pass, OC-SORT made motion estimates robust to occlusion, and BoT-SORT added camera-motion compensation for moving cameras.
• Joint detection and embedding (FairMOT): FairMOT trains one network to output detections and Re-ID embeddings together, so appearance features cost no second model.
• End-to-end tracking (MOTR): MOTR and TrackFormer extend the DETR transformer with track queries that carry each object's identity from frame to frame, so association is learned during training.

Ultralytics YOLO's track mode runs tracking-by-detection on any YOLO26 detection, segmentation, pose, or oriented-box model, with six built-in trackers selected through the tracker argument: TrackTrack (the default), BoT-SORT, ByteTrack, OC-SORT, Deep OC-SORT, and FastTracker. This example tracks the people in a short clip and counts the frames each ID survives:

from collections import Counter

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
video = "https://github.com/ultralytics/assets/releases/download/v0.0.0/decelera_portrait_min.mov"

# Track people (COCO class 0) and count the frames in which each track ID appears
frames_per_id = Counter()
for result in model.track(video, stream=True, classes=[0]):
    if result.boxes.is_track:
        frames_per_id.update(result.boxes.id.int().tolist())

# The five longest trajectories as (track ID, frames)
print(frames_per_id.most_common(5))

Each pair is a track ID and the number of frames it was held, so a long-lived ID means the tracker kept following one person across many frames.

Measuring Tracking Quality#

MOT is scored on benchmarks such as MOT17, MOT20 for dense crowds, and DanceTrack, where dancers look alike and move unpredictably. Three metrics dominate:

• MOTA: One minus the sum of missed objects, false positives, and ID switches divided by the number of ground-truth objects, so it is dominated by detection quality.
• IDF1: The F1 score of correctly identified detections, measuring how consistently each true object keeps one ID.
• HOTA: The geometric mean of detection accuracy and association accuracy, which weighs finding objects and linking them equally.

Challenges#

• Occlusion: An object hidden for many frames can return with a new ID unless motion prediction or appearance matching recovers it.
• Crowds and similar appearance: In dense groups of people or uniformed sports players, IoU and appearance costs become ambiguous and IDs swap.
• Camera motion: A panning or vibrating camera moves every box at once, breaking constant-velocity motion models unless the tracker compensates.
• Real-time budgets: Detection, association, and any Re-ID model share each frame's time budget, and a separate Re-ID model adds a network pass for every detected object.