Visual SLAM (Simultaneous Localization and Mapping)
Visual SLAM builds a map of an unknown space while tracking the camera's pose within it from images. Covers tracking, loop closure, scale, and deep SLAM.
Visual SLAM (simultaneous localization and mapping) is the problem of building a map of an unknown environment while tracking the camera's own position and orientation within it, using images as the main sensor. The two halves depend on each other: locating the camera requires a map, and building the map requires knowing where the camera was. Visual SLAM systems use a single (monocular) camera, a stereo pair, or an RGB-D camera that also measures depth, and many fuse an inertial measurement unit (IMU) as well, which is called visual-inertial SLAM or, without a persistent map, visual-inertial odometry. Phones and headsets use it to anchor augmented reality content, as Apple's ARKit does by fusing camera features with motion-sensor data, and robots and drones use it to navigate where GPS is unavailable.
How Visual SLAM Works#
A typical system runs tracking, mapping, and loop closure as parallel processes. PTAM, a 2007 augmented reality system, separated tracking from mapping this way, and ORB-SLAM3 and most later feature-based systems follow the same design:
- Tracking: For every frame, the front-end detects features such as ORB corners, matches them to the map or the previous frame, and estimates the camera's six-degree-of-freedom pose. On its own, this frame-to-frame estimate is visual odometry. Direct methods such as DSO skip features and align pixel intensities instead.
- Local mapping: Selected keyframes add new 3D points by triangulating matched features, and bundle adjustment jointly refines nearby camera poses and points to minimize reprojection error.
- Loop closure: Small errors accumulate into drift. When the system recognizes a place it has seen before, usually by comparing compact appearance descriptors of keyframes, it adds a constraint between the two visits and optimizes the whole trajectory so the map lines up again.
Motion From Two Views and the Scale Problem#
The core of the tracking step is two-view geometry from epipolar geometry: from features matched between two frames, the essential matrix encodes the camera's rotation and the direction of its translation. This OpenCV example projects synthetic 3D landmarks into two frames with noise and recovers the motion between them:
import cv2
import numpy as np
rng = np.random.default_rng(0)
K = np.array([[500.0, 0, 320], [0, 500.0, 240], [0, 0, 1]]) # camera intrinsics for a 640x480 image
points = rng.uniform([-1.8, -1.4, 3], [1.8, 1.4, 6], (200, 3)) # static 3D landmarks 3-6 m ahead
# Between frames the camera turns 5 degrees and moves 0.3 m to the right
R_true = cv2.Rodrigues(np.array([0.0, np.radians(5), 0.0]))[0]
t_true = np.array([-0.3, 0.0, 0.0])
def project(X, R, t):
"""Project 3D points to pixels and add 0.5 px of feature-detection noise."""
x = (K @ (X @ R.T + t).T).T
return x[:, :2] / x[:, 2:] + rng.normal(0, 0.5, (len(X), 2))
pts1, pts2 = project(points, np.eye(3), np.zeros(3)), project(points, R_true, t_true)
# Two-view motion estimate: essential matrix from matched features, then the relative pose
E, inliers = cv2.findEssentialMat(pts1, pts2, K, method=cv2.RANSAC, threshold=1.0)
_, R, t, _ = cv2.recoverPose(E, pts1, pts2, K, mask=inliers)
print(f"Rotation: {np.degrees(np.linalg.norm(cv2.Rodrigues(R)[0])):.2f} deg (true 5.00)")
print(f"Translation: {t.ravel().round(2).tolist()}, length {np.linalg.norm(t):.2f} (true length 0.30 m)")It recovers a rotation of 5.72 degrees against the true 5.00 and a translation direction of [-1.0, -0.09, 0.02] against the true [-1, 0, 0], but the translation always has length 1. A single camera cannot tell a 0.3 m move past nearby objects from a 3 m move past objects ten times farther away, so monocular SLAM fixes the scale arbitrarily at startup, and the scale drifts over time. A stereo baseline, a depth sensor, or an IMU supplies the metric scale.
Dynamic Scenes and Semantic SLAM#
Most SLAM pipelines assume a static world, so features on walking people and passing cars pull the pose estimate off course. Semantic SLAM systems use segmentation to find movable objects and ignore features on them; DynaSLAM added Mask R-CNN masks to ORB-SLAM2 for this in 2018. An Ultralytics YOLO26 segmentation model does the same job at real-time frame rates, producing a mask that confines feature extraction to the static scene:
import cv2
import numpy as np
from ultralytics import YOLO
model = YOLO("yolo26n-seg.pt")
results = model("https://ultralytics.com/images/bus.jpg", classes=[0, 1, 2, 3, 5, 7]) # people and vehicles
result = results[0]
# Static-scene mask: 255 where SLAM may extract features, 0 on objects that can move
static = np.full(result.orig_shape, 255, np.uint8)
for polygon in result.masks.xy:
cv2.fillPoly(static, [polygon.astype(np.int32)], 0)
orb = cv2.ORB_create()
gray = cv2.cvtColor(result.orig_img, cv2.COLOR_BGR2GRAY)
keypoints = orb.detect(gray, None)
on_moving = sum(static[int(k.pt[1]), int(k.pt[0])] == 0 for k in keypoints)
print(f"{on_moving} of {len(keypoints)} ORB features fall on people or vehicles")
# The SLAM front-end extracts features only where the mask allows
static_keypoints = orb.detect(gray, static)
cv2.imwrite("static_features.jpg", cv2.drawKeypoints(result.orig_img, static_keypoints, None))The printed count is the share of features that would otherwise come from moving objects, and static_features.jpg shows the features left for tracking. Labeled masks also let semantic mapping attach object classes to the map itself.
Visual SLAM vs. Related Techniques#
- LiDAR SLAM: Measures distances directly with laser pulses, so it has metric scale and works in the dark, at higher sensor cost and power. Visual SLAM uses cheap cameras and captures appearance, which helps recognize revisited places, but struggles in low light, motion blur, and textureless scenes such as blank walls.
- Visual odometry: The tracking front-end alone, which estimates motion but keeps no global map and cannot correct drift through loop closure.
- Structure from motion (SfM): Reconstructs a scene offline from an unordered photo collection, whereas SLAM processes frames in order as they arrive.
- Object tracking: Follows specific objects across video frames. SLAM tracks the camera itself relative to the static world.
Classic and Learned Approaches#
Feature-based systems such as ORB-SLAM3, which supports monocular, stereo, RGB-D, and visual-inertial setups, remain widely used because they are fast and accurate in textured scenes. Learned systems replace hand-crafted parts with neural networks: DROID-SLAM (2021) iterates dense optical flow updates through a differentiable bundle adjustment layer. Other recent systems store the map as Gaussian splats or neural radiance fields, producing photorealistic maps that can be rendered from new viewpoints. MASt3R-SLAM, a CVPR 2025 highlight, builds on MASt3R, a learned two-view 3D reconstruction model, to run dense monocular SLAM at 15 FPS without assuming a fixed parametric camera model.










