Semantic Mapping

Semantic mapping is the process of building a spatial representation of an environment and attaching meaningful class labels—such as road, wall, doorway, person, shelf, or vegetation—to locations within it. In AI and robotics, a semantic map answers both “Where is something?” and “What is it?” Instead of representing a scene only as occupied and free space, it provides contextual information that supports navigation, planning, interaction, and scene understanding.

How Semantic Mapping Works#

A semantic mapping system typically combines perception, geometry, localization, and temporal fusion. First, a semantic segmentation model classifies every image pixel. All pixels labeled “road,” for example, form a road region, while pixels labeled “car” identify vehicle regions. The NVIDIA introduction to image segmentation in robotics illustrates how these per-pixel labels support robot perception.

Pixel labels alone remain tied to a two-dimensional camera image. The system needs geometry, often from LiDAR, stereo cameras, or depth estimation, to associate pixels with physical positions. The Open3D RGB-D image workflow demonstrates how registered color and depth images can produce a 3D point cloud.

Accurate projection also depends on camera intrinsics and extrinsics. OpenCV camera calibration documentation explains the parameters used to relate image pixels to 3D points, while the ROS CameraInfo message definition standardizes this calibration information in robotic systems.

Finally, sensor fusion combines observations over time. Robot poses and coordinate transformations place each labeled observation into a shared map frame. ROS tf2 coordinate transforms provide one common mechanism for maintaining these relationships.

Related Concepts and Key Differences#

Semantic mapping overlaps with several computer vision and robotics concepts, but they solve different problems:

• Visual SLAM estimates a camera or robot’s position while constructing a geometric map. Semantic mapping adds class meaning to that geometry. A system may therefore use SLAM for pose estimation and semantic perception for labeling.
• Occupancy mapping describes whether cells are free, occupied, or unknown. The Nav2 mapping and localization guide explains how occupancy grids support path planning. A semantic map can additionally distinguish a wall from a person or a drivable road from a sidewalk.
• Instance segmentation separates individual objects, such as two different cars. Semantic segmentation groups both cars under one pixel class. Semantic maps may use either representation depending on whether object identity must be preserved.
• Semantic search organizes information by conceptual similarity, whereas semantic mapping in robotics associates meanings with physical locations.

Real-World Applications#

Autonomous navigation: A delivery robot can label floors as traversable, walls and curbs as obstacles, and people as dynamic hazards. These labels can be translated into navigation costs so the planner prefers safe surfaces rather than treating every visible region identically. The Nav2 semantic segmentation navigation tutorial shows how class masks and registered point clouds can update a robot’s costmap.

Urban driving: An autonomous vehicle can project road, sidewalk, building, vegetation, pedestrian, and vehicle predictions into a persistent world map. This helps the system understand lane boundaries, recognize non-drivable areas, and reason about where vulnerable road users may appear. The official Cityscapes urban-scene dataset provides densely labeled street imagery for developing this type of scene understanding.

Semantic Perception with Ultralytics YOLO#

The following Ultralytics semantic segmentation workflow produces a dense class map using YOLO26. That output can later be projected into a 2D grid or 3D world coordinate system by a mapping pipeline.

from ultralytics import YOLO

# Load a pretrained semantic segmentation model
model = YOLO("yolo26n-sem.pt")

# Predict a per-pixel class map
results = model("https://ultralytics.com/images/bus.jpg")

# Select the first result and inspect its map dimensions
result = results[0]
class_map = result.semantic_mask.data
print(class_map.shape)

result.save(filename="semantic_scene.jpg")

The semantic_mask contains one class ID per image pixel. It provides the semantic perception layer, while depth, calibration, localization, and temporal fusion are still required to construct a persistent spatial map. Custom pixel-level datasets can be labeled and managed with the Ultralytics Platform annotation tools.

Practical Design Considerations#

Reliable semantic maps require consistent class definitions, accurate sensor calibration, synchronized timestamps, and representative training data. Pose drift can place correct labels in incorrect locations, while segmentation errors may mark obstacles as traversable space. Dynamic objects also require expiration or tracking rules so a parked vehicle or pedestrian does not remain permanently embedded in the map.

Teams should validate both perception accuracy and spatial consistency, test difficult boundaries and small structures, retain prediction confidence where possible, and define how conflicting observations are updated. For safety-critical navigation, semantic information should complement—not automatically replace—geometric obstacle sensing and collision checks.