License YOLO
Back to Ultralytics Glossary

Visual SLAM (Simultaneous Localization and Mapping)

Visual SLAM builds a map of an unknown space while tracking the camera's pose within it from images. Covers tracking, loop closure, scale, and deep SLAM.

Visual SLAM (simultaneous localization and mapping) is the problem of building a map of an unknown environment while tracking the camera's own position and orientation within it, using images as the main sensor. The two halves depend on each other: locating the camera requires a map, and building the map requires knowing where the camera was. Visual SLAM systems use a single (monocular) camera, a stereo pair, or an RGB-D camera that also measures depth, and many fuse an inertial measurement unit (IMU) as well, which is called visual-inertial SLAM or, without a persistent map, visual-inertial odometry. Phones and headsets use it to anchor augmented reality content, as Apple's ARKit does by fusing camera features with motion-sensor data, and robots and drones use it to navigate where GPS is unavailable.

How Visual SLAM Works#

A typical system runs tracking, mapping, and loop closure as parallel processes. PTAM, a 2007 augmented reality system, separated tracking from mapping this way, and ORB-SLAM3 and most later feature-based systems follow the same design:

  • Tracking: For every frame, the front-end detects features such as ORB corners, matches them to the map or the previous frame, and estimates the camera's six-degree-of-freedom pose. On its own, this frame-to-frame estimate is visual odometry. Direct methods such as DSO skip features and align pixel intensities instead.
  • Local mapping: Selected keyframes add new 3D points by triangulating matched features, and bundle adjustment jointly refines nearby camera poses and points to minimize reprojection error.
  • Loop closure: Small errors accumulate into drift. When the system recognizes a place it has seen before, usually by comparing compact appearance descriptors of keyframes, it adds a constraint between the two visits and optimizes the whole trajectory so the map lines up again.

Motion From Two Views and the Scale Problem#

The core of the tracking step is two-view geometry from epipolar geometry: from features matched between two frames, the essential matrix encodes the camera's rotation and the direction of its translation. This OpenCV example projects synthetic 3D landmarks into two frames with noise and recovers the motion between them:

import cv2
import numpy as np

rng = np.random.default_rng(0)
K = np.array([[500.0, 0, 320], [0, 500.0, 240], [0, 0, 1]])  # camera intrinsics for a 640x480 image
points = rng.uniform([-1.8, -1.4, 3], [1.8, 1.4, 6], (200, 3))  # static 3D landmarks 3-6 m ahead

# Between frames the camera turns 5 degrees and moves 0.3 m to the right
R_true = cv2.Rodrigues(np.array([0.0, np.radians(5), 0.0]))[0]
t_true = np.array([-0.3, 0.0, 0.0])


def project(X, R, t):
    """Project 3D points to pixels and add 0.5 px of feature-detection noise."""
    x = (K @ (X @ R.T + t).T).T
    return x[:, :2] / x[:, 2:] + rng.normal(0, 0.5, (len(X), 2))


pts1, pts2 = project(points, np.eye(3), np.zeros(3)), project(points, R_true, t_true)

# Two-view motion estimate: essential matrix from matched features, then the relative pose
E, inliers = cv2.findEssentialMat(pts1, pts2, K, method=cv2.RANSAC, threshold=1.0)
_, R, t, _ = cv2.recoverPose(E, pts1, pts2, K, mask=inliers)
print(f"Rotation: {np.degrees(np.linalg.norm(cv2.Rodrigues(R)[0])):.2f} deg (true 5.00)")
print(f"Translation: {t.ravel().round(2).tolist()}, length {np.linalg.norm(t):.2f} (true length 0.30 m)")

It recovers a rotation of 5.72 degrees against the true 5.00 and a translation direction of [-1.0, -0.09, 0.02] against the true [-1, 0, 0], but the translation always has length 1. A single camera cannot tell a 0.3 m move past nearby objects from a 3 m move past objects ten times farther away, so monocular SLAM fixes the scale arbitrarily at startup, and the scale drifts over time. A stereo baseline, a depth sensor, or an IMU supplies the metric scale.

Dynamic Scenes and Semantic SLAM#

Most SLAM pipelines assume a static world, so features on walking people and passing cars pull the pose estimate off course. Semantic SLAM systems use segmentation to find movable objects and ignore features on them; DynaSLAM added Mask R-CNN masks to ORB-SLAM2 for this in 2018. An Ultralytics YOLO26 segmentation model does the same job at real-time frame rates, producing a mask that confines feature extraction to the static scene:

import cv2
import numpy as np
from ultralytics import YOLO

model = YOLO("yolo26n-seg.pt")
results = model("https://ultralytics.com/images/bus.jpg", classes=[0, 1, 2, 3, 5, 7])  # people and vehicles
result = results[0]

# Static-scene mask: 255 where SLAM may extract features, 0 on objects that can move
static = np.full(result.orig_shape, 255, np.uint8)
for polygon in result.masks.xy:
    cv2.fillPoly(static, [polygon.astype(np.int32)], 0)

orb = cv2.ORB_create()
gray = cv2.cvtColor(result.orig_img, cv2.COLOR_BGR2GRAY)
keypoints = orb.detect(gray, None)
on_moving = sum(static[int(k.pt[1]), int(k.pt[0])] == 0 for k in keypoints)
print(f"{on_moving} of {len(keypoints)} ORB features fall on people or vehicles")

# The SLAM front-end extracts features only where the mask allows
static_keypoints = orb.detect(gray, static)
cv2.imwrite("static_features.jpg", cv2.drawKeypoints(result.orig_img, static_keypoints, None))

The printed count is the share of features that would otherwise come from moving objects, and static_features.jpg shows the features left for tracking. Labeled masks also let semantic mapping attach object classes to the map itself.

  • LiDAR SLAM: Measures distances directly with laser pulses, so it has metric scale and works in the dark, at higher sensor cost and power. Visual SLAM uses cheap cameras and captures appearance, which helps recognize revisited places, but struggles in low light, motion blur, and textureless scenes such as blank walls.
  • Visual odometry: The tracking front-end alone, which estimates motion but keeps no global map and cannot correct drift through loop closure.
  • Structure from motion (SfM): Reconstructs a scene offline from an unordered photo collection, whereas SLAM processes frames in order as they arrive.
  • Object tracking: Follows specific objects across video frames. SLAM tracks the camera itself relative to the static world.

Classic and Learned Approaches#

Feature-based systems such as ORB-SLAM3, which supports monocular, stereo, RGB-D, and visual-inertial setups, remain widely used because they are fast and accurate in textured scenes. Learned systems replace hand-crafted parts with neural networks: DROID-SLAM (2021) iterates dense optical flow updates through a differentiable bundle adjustment layer. Other recent systems store the map as Gaussian splats or neural radiance fields, producing photorealistic maps that can be rendered from new viewpoints. MASt3R-SLAM, a CVPR 2025 highlight, builds on MASt3R, a learned two-view 3D reconstruction model, to run dense monocular SLAM at 15 FPS without assuming a fixed parametric camera model.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning