3D Reconstruction
3D reconstruction recovers the shape of objects and scenes from photos, video, depth cameras, or LiDAR as point clouds, meshes, or radiance fields.
3D reconstruction is the process of recovering the three-dimensional shape of an object or scene from measurements such as photographs, video, depth-camera frames, or LiDAR scans. The output describes where surfaces are in space, as a point cloud of sampled surface points, a polygon mesh, a voxel grid, or a learned function such as a radiance field, often with color or texture attached. Image-based reconstruction, also called photogrammetry, is the computer vision inverse of photography: a camera flattens a 3D world into 2D pixels, and reconstruction uses several views of the same surfaces to undo that projection.
How Image-Based Reconstruction Works#
The classic pipeline starts by detecting distinctive features such as SIFT keypoints in every image and matching them across overlapping photos, keeping only matches consistent with epipolar geometry. Structure from motion then recovers each camera's position and orientation together with a sparse set of 3D points, refining all of them jointly with bundle adjustment, which minimizes the reprojection error of every point in every image. Multi-view stereo uses the known cameras to compute a dense depth map per image, these maps are fused into one point cloud, and surface reconstruction such as Poisson meshing turns the points into a watertight mesh that can be textured from the photos. COLMAP, by Johannes Schönberger and colleagues, is the widely used open-source implementation of this pipeline. Depth cameras and LiDAR skip the matching stage and align overlapping scans directly, typically with the iterative closest point algorithm.
Every 3D point comes from triangulation: given two calibrated cameras, it is the point whose projections best match the pixel where each camera observed it. This example triangulates points on a wall seen by two cameras 0.3 m apart, with half a pixel of noise in each observation:
import cv2
import numpy as np
rng = np.random.default_rng(0)
K = np.array([[800.0, 0, 320], [0, 800.0, 240], [0, 0, 1]]) # camera intrinsics in pixels
# Two calibrated views 0.3 m apart, as projection matrices P = K [R | t]
P1 = K @ np.hstack([np.eye(3), np.zeros((3, 1))])
P2 = K @ np.hstack([np.eye(3), [[-0.3], [0.0], [0.0]]])
for z in [2.0, 10.0]:
# 200 points on a wall z meters away, observed in both images with half-pixel noise
points = np.column_stack([rng.uniform(-1, 1, 200), rng.uniform(-0.5, 0.5, 200), np.full(200, z)])
X = np.vstack([points.T, np.ones(200)])
x1, x2 = P1 @ X, P2 @ X
x1, x2 = x1[:2] / x1[2] + rng.normal(0, 0.5, (2, 200)), x2[:2] / x2[2] + rng.normal(0, 0.5, (2, 200))
# Triangulate: find the 3D point whose projections best match both observations
Xh = cv2.triangulatePoints(P1, P2, x1, x2)
error = np.linalg.norm((Xh[:3] / Xh[3]).T - points, axis=1)
print(f"Wall at {z:.0f} m: median 3D error {100 * np.median(error):.1f} cm")It prints a median error of 0.8 cm for the wall at 2 m and 18.4 cm at 10 m. Five times the distance gives about 23 times the error, close to the quadratic growth that two-view geometry predicts, which is why photogrammetry captures distant surfaces from many viewpoints with wide separation.
Reconstruction assumes a static scene, so people and vehicles moving through a capture produce ghost geometry and mismatched features. COLMAP skips feature extraction wherever a per-image mask is black, and masks of people and cars from an Ultralytics YOLO26 instance segmentation model keep those moving objects out of a static reconstruction.
Representations and Learned Reconstruction#
Point clouds are simple to produce but have no connectivity, meshes give continuous surfaces for measurement, game engines, and digital twins, and voxel grids divide space into cells. KinectFusion, presented by Richard Newcombe and colleagues in 2011, fused live depth-camera frames into a truncated signed distance volume in real time, a volumetric fusion method that libraries such as Open3D still implement. Implicit neural representations store a surface as a network that maps coordinates to occupancy or signed distance, and neural radiance fields and Gaussian splatting model appearance from every direction for novel view synthesis, with geometry that is less directly measurable than a mesh. Feed-forward networks now skip optimization entirely: VGGT, from Oxford's Visual Geometry Group and Meta AI, predicts camera poses, depth maps, and 3D points for one to hundreds of images in a single pass and won the CVPR 2025 Best Paper award.
Related tasks differ in what they deliver. Depth estimation predicts per-pixel distance for one view, a single partial surface. Stereo vision reconstructs from two synchronized cameras. Visual SLAM tracks a moving camera and builds a map in real time for navigation, trading completeness for speed. 3D object detection places boxes around objects instead of recovering their surfaces.
Real-World Applications#
- Augmented reality: Apple's ARKit builds a live mesh of a room on LiDAR-equipped iPhones and iPads, so virtual objects rest on floors and hide behind furniture.
- Manufacturing inspection: Scans of finished parts are compared with their CAD models to measure deviations, and the same scans keep a digital twin of a production line current.
- Robotics: Robots reconstruct the objects they must grasp and the spaces they move through from onboard depth cameras.
- Cultural heritage and mapping: Photogrammetry digitizes buildings, excavations, and whole cities. The 2009 Building Rome in a Day project reconstructed landmarks of Rome from 150,000 Flickr photos in 21 hours on a cluster of 496 processor cores.










