Depth Estimation

Depth estimation is the computer vision task of predicting, for every pixel in an image, the distance from the camera to the surface that pixel shows. The output is a depth map, an image-sized grid of distances that adds the third dimension a flat photo lacks. It gives robots, vehicles, and augmented reality devices the 3D layout of a scene from ordinary cameras.

Ways to Get Depth#

• Stereo vision: Two calibrated cameras a known baseline apart see each point at slightly different horizontal positions. That shift, the disparity, gives depth as focal length × baseline ÷ disparity, so distant objects with small disparities are measured less precisely, and textureless surfaces such as white walls are hard to match between the two images.
• LiDAR and other active sensors: LiDAR measures the return time of laser pulses and gives accurate but sparse points at long range, and time-of-flight and structured-light cameras give dense depth at short range, mostly indoors. These sensors also produce the ground truth that learned models train on.
• Monocular depth estimation: A neural network predicts depth from a single image, using cues such as perspective, the known size of familiar objects, texture gradients, and occlusion. The problem is ill-posed, since a toy car close up and a real car far away can produce the same image, so the model relies on what it learned about typical scenes.
• Visual SLAM and other multi-view methods: Structure from motion and visual SLAM recover depth by triangulating points across frames taken from different positions.

Monocular Depth Estimation with Deep Learning#

Learned depth models come in two kinds. Relative depth models predict the order and shape of surfaces up to an unknown scale and shift, which is enough for effects such as background blur and for ranking what is closer. MiDaS made this approach work across very different scenes by training with a scale- and shift-invariant loss on a mix of datasets, including 3D films, and Depth Anything V2 trains models from 25 million to 1.3 billion parameters on synthetic labels and large sets of pseudo-labeled real images. Metric depth models predict distances in meters, which robots and measurement tools need: Apple's Depth Pro produces a 2.25-megapixel metric depth map in 0.3 seconds on a standard GPU without camera intrinsics.

Ultralytics YOLO26 depth models predict metric depth from one image with a log-depth head whose output spans about 0.02 to 150 meters, so the same model covers rooms a few meters deep and driving scenes out to 80 meters. Loading one, predicting, and saving the result takes a few lines:

from ultralytics import YOLO

# Load a YOLO26 monocular depth model
model = YOLO("yolo26n-depth.pt")

# Predict a depth map for one image
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Per-pixel depth in meters, at the original image's resolution
depth = result.depth.data
print(f"Nearest {depth.min():.1f} m, farthest {depth.max():.1f} m")

# Save a colorized depth map blended over the photo
result.save(filename="depth_result.jpg")

The printed range shows the nearest and farthest surfaces the model sees in the street scene, and the saved image colors each pixel by its predicted distance.

Measuring Accuracy#

Depth models are scored per pixel against sensor ground truth. Absolute relative error (AbsRel) averages the error as a fraction of the true depth, root mean squared error (RMSE) reports it in meters, and δ₁ is the share of pixels whose prediction is within a factor of 1.25 of the truth. The standard benchmarks are NYU Depth V2, indoor scenes captured with a Kinect sensor, and KITTI, driving scenes with LiDAR ground truth, usually scored up to 80 meters. Relative depth predictions are aligned to the ground truth in scale and shift before scoring, so their numbers measure shape only. Averages over all pixels also hide depth bleeding, where predicted depth smears across object boundaries, which is why Depth Pro introduced dedicated boundary-accuracy metrics.

Depth Estimation vs. Related Tasks#

• 3D object detection: Predicts a 3D box with position, size, and heading for each object, often using depth as an input.
• Semantic segmentation: Assigns each pixel a class label, and combined with a depth map it tells a robot both what each surface is and how far away it is.
• 3D reconstruction: Builds a full mesh or point cloud of a scene, usually by fusing depth from many viewpoints.

Real-World Applications#

• Driver assistance and autonomous driving: Camera-based depth adds distance estimates to detected cars and pedestrians, alongside or in place of LiDAR.
• Robotics: Grasping and navigation need the distance to each surface, and depth from a single RGB camera avoids extra sensors on small robots.
• Augmented reality: Depth maps let virtual objects be hidden correctly behind real ones and placed on real surfaces.
• Photography: Phone portrait modes blur the background using depth from dual cameras, dual-pixel sensors, or a monocular model, with blur strength set by each pixel's estimated distance.