Semantic Mapping

Semantic mapping is the process of building a spatial map of an environment in which every location carries a meaning, a class label such as road, sidewalk, wall, door, chair, or person, on top of its geometry. A geometric map tells a robot where surfaces are, and a semantic map also tells it what they are, so it can follow a request like "go to the kitchen table", keep off the lawn, or expect pedestrians near a crosswalk. Semantic maps combine semantic segmentation or object detection with depth sensing and localization, and they are a core part of perception for robotics, autonomous vehicles, and spatial computing.

How Semantic Mapping Works#

Each new camera frame goes through three steps. A segmentation model labels every pixel with a class. Depth from LiDAR, a stereo or RGB-D camera, or monocular depth estimation lifts the labeled pixels into 3D points, and the robot's pose, estimated by visual SLAM or odometry, places those points into the cells of a shared map. Finally, sensor fusion over time merges the new labels with everything each cell has accumulated so far, and moving objects get separate handling so that a person walking by does not become a permanent wall.

Ultralytics YOLO26 semantic segmentation models pretrained on Cityscapes label every pixel with one of 19 urban classes, including road, sidewalk, building, vegetation, person, and car, the per-frame input a street-level semantic map fuses.

Fusion is what turns noisy per-frame predictions into a reliable map. SemanticFusion, by John McCormac and colleagues at Imperial College London in 2017, kept a probability distribution over classes for every surface element and multiplied in each new network prediction, a recursive Bayesian update. This simulation applies the same update to a map of 1,000 cells observed by a segmentation model that is right 70% of the time:

import numpy as np

rng = np.random.default_rng(0)
num_cells, num_classes = 1000, 4  # map cells; classes such as road, sidewalk, building, vegetation
truth = rng.integers(0, num_classes, num_cells)

# Sensor model: segmentation labels a cell correctly 70% of the time, otherwise picks another class
likelihood = np.full((num_classes, num_classes), 0.1)
np.fill_diagonal(likelihood, 0.7)
log_belief = np.zeros((num_cells, num_classes))  # uniform prior in every cell

for frame in range(1, 11):
    # A new frame observes every cell from a new viewpoint, with independent errors
    wrong = rng.random(num_cells) > 0.7
    labels = np.where(wrong, (truth + rng.integers(1, num_classes, num_cells)) % num_classes, truth)

    # Recursive Bayesian update: add the log-likelihood of this observation to each cell's belief
    log_belief += np.log(likelihood[labels])
    if frame in (1, 3, 10):
        print(f"Frame {frame:>2}: {(log_belief.argmax(axis=1) == truth).mean():.1%} of cells correct")

The map is 69.6% correct after one frame, 80.6% after three, and 98.2% after ten. The gain depends on errors changing between observations, so fusion helps most when the robot sees each place from varied viewpoints, while a mistake the model repeats from every angle stays in the map. SemanticFusion reported the same pattern, with a larger improvement over single-frame labels on a dataset with more viewpoint variation.

Map Representations and Related Concepts#

Semantic maps range from 2D grids, where each cell of an occupancy map also stores a class, to labeled 3D point clouds, voxels, and meshes. MIT's Kimera builds a metric-semantic 3D mesh in real time, and 3D scene graphs organize the same information into objects, rooms, and buildings linked by relations. Open-vocabulary maps store CLIP embeddings instead of fixed classes, so a robot can search its map for any text query, such as "fire extinguisher".

Semantic mapping builds on several related tasks. Visual SLAM supplies the camera poses and geometric map. Occupancy mapping records only whether space is free, occupied, or unknown. Instance segmentation and panoptic segmentation separate individual objects, which object-level maps need to count chairs or track parked cars. Semantic search is an unrelated retrieval technique that shares only the word.

Real-World Applications#

• Autonomous driving: High-definition maps record lanes, crosswalks, traffic lights, and curbs at centimeter-level accuracy, and online semantic mapping builds a bird's-eye-view map of the same elements from the car's own sensors.
• Mobile robots: The ROS 2 navigation stack, Nav2, offers a semantic segmentation costmap layer that assigns a traversal cost to each class, so a robot plans paths over preferred surfaces.
• Agriculture: Field robots and drones map crops and weeds row by row for targeted spraying and yield estimates.
• Augmented reality: Apple's ARKit classifies each face of its reconstructed room mesh as wall, floor, ceiling, table, seat, window, door, or none.