Optical Flow

Optical flow is the apparent motion of brightness patterns between two images, usually consecutive video frames. An optical flow algorithm estimates, for each pixel in the first frame, a 2D displacement vector (dx, dy) that points to where that pixel's content appears in the second frame. The result, called a flow field, captures motion from moving objects and from the camera itself, which lets computer vision systems stabilize video, estimate ego-motion, interpolate frames, and recognize actions. Flow measures what the camera sees rather than true 3D motion: a perfectly uniform sphere spinning in place produces no flow, and a moving shadow produces flow where nothing physical moves.

How Optical Flow Is Estimated#

Most methods start from brightness constancy, the assumption that a point keeps the same intensity as it moves. Linearizing it gives one equation per pixel, Ix·u + Iy·v + It = 0, where Ix, Iy, and It are the image's derivatives in x, y, and time, and (u, v) is the unknown flow. One equation cannot determine two unknowns, which is the aperture problem: looking at a straight edge through a small window, only the motion perpendicular to the edge is measurable. Classic methods add a second assumption to resolve it:

• Lucas-Kanade method (1981): Assumes the flow is constant within a small window and solves the equations of all its pixels by least squares. It works where the window contains gradients in two directions, such as corners, so it is usually run as sparse flow on selected feature points.
• Horn-Schunck method (1981): Assumes the flow field is smooth across the whole image and solves a global optimization, producing dense flow for every pixel.

Both assumptions break for large motions, so practical methods estimate flow coarse to fine on an image pyramid, where a 16-pixel motion at full resolution is a 2-pixel motion three levels down:

OpenCV implements pyramidal Lucas-Kanade for sparse points and Gunnar Farnebäck's 2003 polynomial expansion algorithm for dense flow. This code moves a random texture by a known amount and recovers the motion with dense Farnebäck flow:

import cv2
import numpy as np

# A smooth random texture, then the same texture moved 3 px right and 2 px down
rng = np.random.default_rng(0)
texture = cv2.GaussianBlur(rng.random((240, 320)), (0, 0), 3)
frame1 = cv2.normalize(texture, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
frame2 = np.roll(frame1, shift=(2, 3), axis=(0, 1))

# Dense Farneback optical flow: one (dx, dy) displacement vector for every pixel
flow = cv2.calcOpticalFlowFarneback(
    frame1, frame2, None, pyr_scale=0.5, levels=3, winsize=15, iterations=3, poly_n=5, poly_sigma=1.2, flags=0
)
print(f"flow field shape: {flow.shape}")
print(f"median motion: dx={np.median(flow[..., 0]):.2f}, dy={np.median(flow[..., 1]):.2f} pixels")

It prints flow field shape: (240, 320, 2) and median motion: dx=3.00, dy=2.00 pixels: one two-component vector for each of the 76,800 pixels, recovering the 3-pixel horizontal and 2-pixel vertical shift.

Deep Learning Methods#

FlowNet showed in 2015 that a convolutional neural network could learn flow end to end from synthetic training data, since real video seldom comes with ground-truth flow. PWC-Net built the pyramid, warping, and cost-volume ideas of classic methods into a compact network. RAFT, which won the ECCV 2020 best paper award, compares every pixel of the first frame with every pixel of the second in a correlation volume and refines a single high-resolution flow field with repeated recurrent updates. Learned methods handle large displacements, motion blur, and textureless regions far better than classic ones, and torchvision ships pretrained RAFT models. Accuracy is reported as endpoint error (EPE), the average distance in pixels between predicted and true flow vectors, on benchmarks such as MPI Sintel and KITTI 2015.

Optical Flow vs. Related Terms#

• Object tracking: Follows whole objects with persistent IDs across many frames, while optical flow describes per-pixel motion between two frames and knows nothing about objects. Trackers use flow as a cue: in Ultralytics YOLO track mode, the default TrackTrack tracker and BoT-SORT estimate camera motion with sparse Lucas-Kanade flow (gmc_method: sparseOptFlow), tracking up to 400 corners in a half-resolution frame, fitting a rotation, scale, and translation to them with RANSAC, and shifting every predicted track box by that motion before matching it to new detections.
• Point tracking: Follows chosen points through long videos, including through occlusions, rather than estimating motion for every pixel between two frames.
• Scene flow: The 3D motion of points in space, estimated from stereo or depth data. Optical flow is its projection onto the image plane.
• Video codec motion vectors: H.264, HEVC, and AV1 encoders store one motion vector per block, chosen to minimize the bits needed to encode the frame, so it often departs from the true motion.

Real-World Applications#

• Video stabilization: The dominant flow across a frame reveals camera shake, which software cancels by warping each frame in the opposite direction.
• Frame interpolation: Slow-motion and frame-rate conversion synthesize in-between frames by moving pixels partway along their flow vectors.
• Autonomous vehicles and drones: Flow supports visual odometry, and the rate at which an object's flow expands warns of an approaching collision.
• Action recognition: The two-stream network of 2014 classified actions from a spatial stream of single frames and a temporal stream fed with a stack of 10 consecutive optical flow fields.