Scene Flow

Scene flow is the three-dimensional motion field of a dynamic scene over time. It describes how visible points move in real-world 3D space between frames, including sideways, vertical, and toward-or-away-from-camera motion. In computer vision, scene flow helps systems understand not only what appears to move in an image, but how the surrounding geometry actually changes.

How Scene Flow Represents 3D Motion#

A scene flow estimate assigns a 3D displacement or velocity vector to each observed scene point. For example, if a pedestrian walks across a road while approaching a camera, the corresponding vectors describe both the lateral motion and the decreasing distance to the camera.

This differs from optical flow, which represents apparent pixel movement across the two-dimensional image plane. As the Carnegie Mellon Scene Flow project explains, optical flow can be understood as the projection of 3D scene flow onto a camera image. Consequently, two points may have similar image motion while moving differently in depth.

Dense scene flow estimates motion for most visible points, while sparse scene flow covers selected features, tracked points, or LiDAR returns. Dense output provides richer geometric detail but requires more computation and reliable correspondence between frames.

Scene Flow vs. Related Vision Concepts#

Scene flow combines spatial structure with temporal motion, placing it between several related tasks:

• Optical flow estimation with OpenCV: Estimates horizontal and vertical pixel displacement. It cannot independently determine whether an object moved through depth or whether the camera caused the apparent motion.
• Depth estimation: Determines how far surfaces are from the camera, usually for one frame. Scene flow additionally describes how those 3D positions change over time.
• Stereo vision: Uses corresponding pixels from synchronized cameras to recover depth. The OpenCV stereo depth workflow explains how disparity between views supports 3D reconstruction.
• Object tracking: Maintains object identities across frames, commonly using bounding boxes or masks. Scene flow instead estimates motion at the point or pixel level and can represent different motions within one deforming object.
• Point-cloud motion: LiDAR and RGB-D systems may estimate scene flow directly between 3D point sets. The Open3D point-cloud guide introduces the geometric representation used by many such pipelines.

Scene flow also depends on camera geometry. Accurate OpenCV camera calibration helps separate actual scene motion from changes caused by camera rotation, translation, or lens distortion.

Real-World Applications#

• Autonomous driving: A perception system can estimate whether a nearby vehicle is changing lanes, approaching rapidly, or moving with traffic. Unlike 2D motion alone, scene flow supports time-to-collision reasoning because it includes movement along the depth axis. It can complement detection, tracking, and depth components in automotive computer vision solutions. The KITTI scene flow benchmark illustrates evaluation on road scenes containing camera motion and independently moving objects, while NHTSA automated vehicle safety guidance provides broader context for safety-focused perception systems.

• Robotics: A mobile robot can use scene flow to distinguish a stationary shelf from a moving worker, estimate an obstacle’s 3D trajectory, and update its path before a collision occurs. In robotics vision solutions, this information may be combined with cameras, depth sensors, and LiDAR. The ROS sensor message documentation defines common interfaces for exchanging images, camera information, and point clouds between robotic components.

Estimation Challenges and Data#

Scene flow can be derived from stereo video, RGB-D cameras, LiDAR sequences, or monocular video with learned depth and motion cues. Stereo and active-depth sensors provide stronger geometric measurements, whereas monocular systems must infer scale and resolve ambiguities from visual context.

Common failure cases include occlusion, motion blur, reflective surfaces, textureless regions, thin structures, changing illumination, and fast movement between frames. Camera motion creates an additional challenge because the system must separate ego-motion from independently moving objects. Errors can produce incorrect 3D trajectories, causing downstream navigation or collision-prediction systems to misjudge speed and distance.

Training data may include RGB frames, depth or disparity, optical flow, segmentation, and camera parameters. The Freiburg Scene Flow datasets demonstrate how these complementary labels can describe geometry and motion together.

Practical Workflow#

Ultralytics YOLO26 can provide the depth component of a scene flow pipeline through its documented monocular depth estimation task. The following example produces a dense depth map for one frame:

from ultralytics import YOLO

# Estimate the scene's per-pixel 3D structure.
model = YOLO("yolo26n-depth.pt")
results = model("https://ultralytics.com/images/bus.jpg")

result = results[0]
result.save(filename="depth_result.jpg")

print(result.depth.data.shape)

This YOLO26 workflow does not calculate scene flow by itself. It supplies per-pixel depth, which a complete pipeline can combine with consecutive frames, temporal correspondences, and calibrated camera poses to estimate 3D displacement. In practice, teams should validate each component separately before evaluating the full motion field, especially around occlusions and depth boundaries.