Semantic Segmentation

Semantic segmentation is a computer vision task that assigns a class label to every pixel in an image. The output is a dense class map with the same height and width as the input, where each value is a class ID such as road, sidewalk, building, sky, or person. Where image classification gives one label per image and object detection draws a box per object, semantic segmentation describes the whole scene region by region, including amorphous background areas like grass and sky that have no countable instances.

How Semantic Segmentation Works#

A semantic segmentation model is a per-pixel classifier. A convolutional neural network (CNN) or vision transformer outputs one score map per class, usually at reduced resolution; the maps are upsampled to the image size and each pixel takes the class with the highest score. Training compares every pixel's prediction with a labeled mask using cross-entropy, and pixels marked with an ignore value, often 255, are left out of the loss.

The core difficulty is that recognizing what a region is requires a large receptive field and heavy downsampling, while drawing accurate boundaries requires fine spatial detail. Fully convolutional networks (FCN) first turned classification networks into dense predictors in 2014. Encoder-decoder designs such as U-Net then recovered boundaries by passing high-resolution encoder features to the decoder through skip connections, and DeepLab used dilated (atrous) convolutions to widen context without shrinking the feature map. Real-time models fuse coarse context with fine detail like this:

Ultralytics YOLO26 semantic segmentation models, marked by the -sem suffix, follow this pattern: they classify pixels from 1/8-scale neck features that already carry fused context, then upsample the logits 8× to the input size. They come pretrained on the 19 Cityscapes classes or the 150 ADE20K classes and train on mask images or on YOLO polygon labels converted to masks on the fly.

from ultralytics import YOLO

# Load a YOLO26 semantic segmentation model trained on the 19 Cityscapes classes
model = YOLO("yolo26n-sem.pt")

# Predict one class ID for every pixel of a street image
results = model("https://ultralytics.com/images/bus.jpg")
class_map = results[0].semantic_mask.data

# Measure how much of the image each class covers
ids, counts = class_map.unique(return_counts=True)
for class_id, count in zip(ids.tolist(), counts.tolist()):
    print(f"{results[0].names[class_id]}: {count / class_map.numel():.1%}")

The printout lists each class present and its share of the image. All the people in the photo fall into a single person share, because the class map stores a class per pixel and nothing about which object a pixel belongs to.

Semantic vs. Instance and Panoptic Segmentation#

Semantic segmentation is one of three image segmentation tasks:

• Instance segmentation: Gives each countable object its own mask and ID, so three cars become car 1, car 2, and car 3, and leaves background pixels unlabeled.
• Panoptic segmentation: Labels every pixel with a class, as semantic segmentation does, and adds an instance ID to pixels of countable objects.

Semantic segmentation suits scenes where regions matter more than object counts, such as drivable road area or land cover.

For help choosing between semantic and instance segmentation, see the semantic vs. instance segmentation guide, and the Ultralytics Academy lesson Semantic Segmentation: Label Every Pixel walks through labeling data and training a model for the task.

Real-World Applications#

• Autonomous vehicles: Driving stacks segment camera frames into road, sidewalk, lane markings, vehicles, and pedestrians to find drivable space and obstacles.
• Medical imaging: Models outline organs, tumors, and lesions in CT and MRI scans so clinicians can measure volumes and plan treatment; U-Net was originally designed for biomedical images.
• Satellite image analysis: Land-cover maps label every pixel as water, forest, cropland, or built-up area to track deforestation and urban growth.
• Agriculture: Field robots and drones classify pixels as crop, weed, or soil so sprayers treat only the weeds.

Benchmarks and Metrics#

Models are evaluated with mean Intersection over Union (IoU), or mIoU: the overlap between predicted and true pixels is computed for each class and then averaged, so rare classes such as poles and traffic signs weigh as much as road and sky. ADE20K covers 150 indoor and outdoor classes for general scene parsing, and Cityscapes, the best-known driving benchmark, contains 5,000 street images from 50 cities with fine pixel-level labels, ranked by mIoU over 19 evaluation classes.