Scene Flow

Scene flow is the three-dimensional motion field of a scene: for every visible surface point, a 3D vector from where the point is in one frame to where it is in the next. Sundar Vedula, Simon Baker, and colleagues at Carnegie Mellon University introduced the term in their 1999 paper Three-Dimensional Scene Flow, defining it as the 3D counterpart of optical flow. Scene flow combines geometry and motion, so it tells a computer vision system both where each point is and how fast it is moving in meters, including toward or away from the camera. That makes it a core signal for autonomous vehicles and robotics, where a car pulling out or a person stepping into a path has to be recognized from its motion.

Scene Flow vs. Optical Flow#

Optical flow is the projection of scene flow onto the image plane: it records how far each pixel moves sideways and up or down, and nothing about motion along the camera's line of sight. Two objects with identical optical flow can therefore move very differently. Scene flow recovers the missing component with depth estimation at both times, back-projecting each pixel along its viewing ray to its measured depth:

import numpy as np

K_inv = np.linalg.inv(np.array([[500.0, 0, 320], [0, 500.0, 240], [0, 0, 1]]))  # inverse camera intrinsics

# Two objects with identical optical flow: each moves from pixel (400, 240) to (410, 240) in 0.1 s
for name, depth_before, depth_after in [("cyclist crossing", 8.0, 8.0), ("car approaching", 8.0, 6.0)]:
    # Back-project each pixel along its viewing ray to the measured depth
    before = depth_before * K_inv @ np.array([400, 240, 1.0])
    after = depth_after * K_inv @ np.array([410, 240, 1.0])

    # Scene flow is the 3D displacement of the same surface point
    flow = after - before
    print(f"{name}: scene flow {flow.round(2)} m, speed {np.linalg.norm(flow) / 0.1:.1f} m/s")

Both objects shift 10 pixels to the right, yet the cyclist moves 0.16 m sideways at 1.6 m/s while the car closes 2 m in a tenth of a second at 20.1 m/s. At 6 m away and approaching at 20 m/s, the car reaches the camera in 0.3 seconds.

Scene flow also differs from object tracking, which follows object identities as bounding boxes, and from point tracking, which follows chosen 2D points across many frames. Ultralytics YOLO speed estimation works at the tracking level in 2D: it follows each tracked box and converts its pixel displacement to km/h with one meters-per-pixel scale for the whole view, which holds when the camera looks down on a road and traffic moves parallel to the image.

How Scene Flow Is Estimated#

The input decides the method. A stereo camera gives disparity at both times plus optical flow between frames, the setup of the KITTI scene flow benchmark that Moritz Menze and Andreas Geiger released in 2015 with 200 training and 200 test road scenes. RGB-D cameras and LiDAR give two point clouds, often merged with camera images through sensor fusion, and networks such as FlowNet3D learn to match points between them directly. A single camera relies on learned monocular depth and optical flow for each pair of frames.

When the camera moves, every static point appears to move too, so a full system also estimates ego-motion, usually from visual SLAM or odometry, and treats points whose flow disagrees with it as independently moving. Driving scenes consist mostly of rigid bodies, so methods such as Menze and Geiger's object scene flow and RAFT-3D group pixels into a few rigidly moving objects, which stabilizes estimates on textureless car bodies. Occlusions are the hardest case, since points that appear or disappear between frames have no match to measure.

Deep learning models need dense ground truth that real sensors cannot capture, so training relies heavily on synthetic data. The Freiburg scene flow datasets, including FlyingThings3D, render tens of thousands of stereo frames with exact disparity, optical flow, and disparity change, and FlowNet3D, trained only on FlyingThings3D, transferred to real KITTI LiDAR scans.

Real-World Applications#

• Mobile robots: Robots sharing warehouses and sidewalks with people use the 3D motion of nearby obstacles to predict where they will be a second later and replan around them.
• Dynamic scene reconstruction: 4D Gaussian splatting and dynamic neural radiance fields model a moving scene by learning how each point or Gaussian moves over time, a learned scene flow that lets a recorded video be replayed from new viewpoints.
• Autonomous driving: LiDAR scene flow separates moving vehicles and pedestrians from the static world between consecutive sweeps. The Waymo Open Dataset added scene flow labels in 2021, with about 1,000 times more annotated frames than earlier real-world scene flow datasets.