Point Tracking

Point tracking is the computer vision task of following individual points through a video: given a query point in one frame, such as a spot on a sleeve or the corner of a window, a tracker estimates where that same physical point appears in every other frame and whether it is visible. Recent research calls it tracking any point (TAP), because the query can be any pixel instead of a predefined keypoint. Point tracking follows a sparse set of points across many frames, including through occlusion, which sets it apart from object tracking, which follows whole objects as bounding boxes with IDs, and from optical flow, which estimates the motion of every pixel between two consecutive frames.

How Point Tracking Works#

The classical Kanade-Lucas-Tomasi (KLT) tracker picks corners with strong texture in two directions, the Shi-Tomasi "good features to track," and finds each one in the next frame by solving for the small shift that best aligns the patch around it, using an image pyramid to handle larger motion. OpenCV implements both steps:

import cv2
import numpy as np

# Read an overhead clip of people walking and pick 300 corners in its first frame
cap = cv2.VideoCapture("https://github.com/ultralytics/assets/releases/download/v0.0.0/solutions_ci_demo.mp4")
prev = cv2.cvtColor(cap.read()[1], cv2.COLOR_BGR2GRAY)
points = cv2.goodFeaturesToTrack(prev, maxCorners=300, qualityLevel=0.01, minDistance=5)
start = points.copy()

# Follow every point frame to frame with pyramidal Lucas-Kanade optical flow
while (frame := cap.read()[1]) is not None:
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    points, status, _ = cv2.calcOpticalFlowPyrLK(prev, gray, points, None)
    found = status.ravel() == 1  # keep only the points found again
    points, start, prev = points[found], start[found], gray

distance = np.linalg.norm(points - start, axis=-1)
print(f"{len(points)} of 300 points tracked to the last frame, median shift {np.median(distance):.1f} px")

It prints 292 of 300 points tracked to the last frame, median shift 28.7 px: through the clip's 62 frames the tracker loses 8 points, and the rest end a median of 28.7 pixels from where they started. KLT compares only consecutive frames, so small errors add up, and it has no notion of occlusion: a point on the floor that a passerby covers can slide onto the passerby and keep being reported.

Learned Point Trackers#

Deep learning trackers reason over many frames at once and predict a position and a visibility flag for every frame. The TAP-Vid benchmark defined the task in 2022 with real videos whose point tracks people labeled by hand. PIPs (2022) refines a point's trajectory over windows of frames so it can bridge short occlusions, and TAPIR (2023) first matches the query point's features against every frame, then refines the track with temporal context. Meta's CoTracker tracks many points jointly with a transformer, so points on the same object support each other when one of them is hidden. Labeled trajectories in real videos are scarce, so CoTracker3 (2024) trains on synthetic videos and then on real videos pseudo-labeled by other trackers, and it reports better results than previous trackers trained on 1,000 times more data.

Real-World Applications#

Ultralytics track mode uses point tracking to cancel camera motion in multi-object tracking: its default TrackTrack tracker and BoT-SORT pick up to 400 corners per frame, follow them with pyramidal Lucas-Kanade flow, and fit a rotation, scale, and translation with RANSAC (gmc_method: sparseOptFlow), then shift every track's predicted box before matching it to the new YOLO26 detections.

• Visual effects: Matchmoving tracks points on a scene so inserted graphics stay locked to surfaces, and stabilization cancels the camera motion that tracked points reveal.
• 3D reconstruction: Structure from motion and visual SLAM triangulate points tracked across frames to recover camera poses and scene geometry.
• Robot learning: Any-point Trajectory Modeling predicts how points in the scene will move during a task and turns those tracks into robot actions, outperforming video-pretraining baselines by about 80% on average across more than 130 tasks.