Object Tracking

Object tracking is a computer vision task that locates objects in successive video frames and links those locations into trajectories, giving each object a persistent ID. A car that enters the frame as ID 7 stays ID 7 while it moves, turns, and passes briefly behind a bus, which lets a system count it once, measure its speed, and reason about where it is going. Single-object tracking follows one target chosen in the first frame, while multi-object tracking (MOT), the most common form in practice, follows every object of the classes of interest at once.

How Object Tracking Works#

Most multi-object trackers use tracking-by-detection: an object detection model finds objects in each frame, and a tracker links those detections over time. The tracker's tools are a motion model, usually a Kalman filter, box overlap measured by Intersection over Union (IoU), optional appearance embeddings from an object re-identification (Re-ID) model, and a linear assignment solver such as the Hungarian algorithm. Every frame, it runs four steps:

1. Predict: The Kalman filter, assuming constant velocity, predicts where each existing track's box will appear in the new frame.
2. Score: The tracker builds a cost matrix between predicted tracks and new detections from IoU, motion, and, in some trackers, appearance similarity.
3. Assign: The solver picks the lowest-cost one-to-one matches.
4. Manage tracks: Matched tracks update their state, confident unmatched detections start new tracks, and unmatched tracks are kept as "lost" for a buffer of frames so they can be recovered after an occlusion.

The lifecycle step is what keeps IDs stable when detections drop out for a few frames:

The core design dates to SORT in 2016, which used only a Kalman filter, IoU costs, and the Hungarian algorithm. DeepSORT added appearance features to recover identities after occlusion, and ByteTrack improved recall by also matching low-confidence detections, which are often partly occluded objects, in a second pass. Later trackers add camera-motion compensation for moving cameras and richer association rules for crowds.

Tracking vs. Detection#

A detector is stateless: it processes each frame independently, so the car in frame 1 and the car in frame 2 are unrelated detections. Tracking adds state that carries identities from frame to frame, which counting, dwell time, and speed estimation all depend on. Tracking also differs from point tracking and optical flow, which follow individual pixels or surface points without grouping them into labeled objects.

Ultralytics YOLO's track mode adds this state to any YOLO26 detection, segmentation, pose, or oriented-box model, with six built-in trackers including ByteTrack, BoT-SORT, and the default, TrackTrack:

from ultralytics import YOLO

# Load a YOLO26 detector; track() pairs it with a tracker that assigns persistent IDs
model = YOLO("yolo26n.pt")

detections, track_ids = 0, set()
for result in model.track(
    "https://github.com/ultralytics/assets/releases/download/v0.0.0/decelera_portrait_min.mov", stream=True
):
    detections += len(result.boxes)
    if result.boxes.is_track:
        track_ids.update(result.boxes.id.int().tolist())

print(f"Detections summed over all frames: {detections}")
print(f"Unique tracked objects: {len(track_ids)}")

The unique ID count is smaller than the summed detections because each ID stands for one object across every frame in which it appears.

Real-World Applications#

• Traffic monitoring and speed estimation: Tracking each vehicle across frames supports counting, lane-level flow analysis, and speed estimates from the distance a track covers per second.
• Retail analytics: Store cameras track shoppers to measure footfall, queue length, and dwell time in front of displays without counting the same person twice.
• Sports analysis: Broadcasters and coaches track players and the ball to compute distance covered, speed, and team formations.
• Autonomous vehicles and robotics: Perception stacks track pedestrians and vehicles to estimate their velocities and predict where they will be in the next second.

Measuring Tracking Quality#

Trackers are compared on benchmarks such as MOT17, MOT20 for dense crowds, and DanceTrack for people who look alike and move unpredictably. MOTA sums misses, false positives, and identity switches into one score dominated by detection quality, IDF1 measures how consistently each true object keeps a single ID, and HOTA balances both as the geometric mean of detection accuracy and association accuracy.