Image Segmentation

Image segmentation is the computer vision task of dividing an image into regions by labeling its pixels. Its output is a set of masks that show exactly which pixels belong to which class or object. Compared with object detection, which locates each object with a rectangular bounding box that also contains background, segmentation traces each object's or region's true outline, and it is far more detailed than image classification, which assigns one label to the entire image. That precision matters whenever area, outline, or contact between objects drives a decision, from measuring a tumor to finding the free space in front of a car.

Types of Image Segmentation#

Segmentation tasks differ in what a pixel label means. Computer vision distinguishes "things," countable objects such as people and cars, from "stuff," amorphous regions such as road, grass, and sky.

• Semantic segmentation: Assigns every pixel a class, covering both things and stuff, and merges all objects of one class into one region.
• Instance segmentation: Detects each countable object and gives it its own mask and ID, so two touching people stay separate; background pixels stay unlabeled.
• Panoptic segmentation: Assigns every pixel a class and adds an instance ID to pixels of countable objects, unifying the other two.
• Interactive segmentation: Returns a mask for whatever a user's click, box, or text prompt indicates, the way promptable models such as the Segment Anything Model (SAM) work.

The first three answer different questions, which makes choosing between them a short decision:

How Segmentation Models Work#

Classical methods group pixels by low-level cues such as intensity, color, and edges: thresholding with Otsu's method, region growing, the watershed transform, k-means color clustering, and graph cuts. OpenCV implements several of them, and they remain useful on clean, controlled images such as backlit parts on a conveyor; they group pixels without knowing what each group represents.

Deep learning models learn both the boundaries and the meaning of regions from labeled masks. Encoder-decoder networks such as U-Net downsample an image to capture context and upsample it back to full resolution for per-pixel predictions. Instance models such as Mask R-CNN and YOLO segmentation models attach a mask head to an object detector, and transformer models such as Mask2Former handle semantic, instance, and panoptic segmentation with a single architecture. Promptable foundation models have extended segmentation to arbitrary objects: SAM arrived in 2023 after training on 1.1 billion masks from 11 million images, SAM 2 added video in 2024, and SAM 3 followed in November 2025, segmenting every instance of a concept named in a text prompt. Because segmentation labels are polygons or per-pixel masks that take far longer to draw than boxes, these promptable models now draft many training masks from a single click.

The task type also fixes the shape of the output. Ultralytics YOLO26 ships both as separate model families, -seg for instance and -sem for semantic segmentation, and running them on the same photo shows the difference:

from ultralytics import YOLO

# Load an instance segmentation model and a semantic segmentation model
instance_model = YOLO("yolo26n-seg.pt")
semantic_model = YOLO("yolo26n-sem.pt")

# Segment the same street image with both
image = "https://ultralytics.com/images/bus.jpg"
instance_results = instance_model(image)
semantic_results = semantic_model(image)

# Instance output: one binary mask per detected object, shaped (objects, height, width)
print(instance_results[0].masks.data.shape)

# Semantic output: one class map for the whole image, shaped (height, width)
print(semantic_results[0].semantic_mask.data.shape)

The first dimension of the instance output equals the number of detected objects, which is what lets instance results be counted. Each output type has its own metric: mask mean average precision (mAP) for instance segmentation, mean Intersection over Union (IoU) for semantic segmentation, and panoptic quality (PQ) for panoptic segmentation.

Real-World Applications#

• Medical image analysis: Radiology tools outline organs, tumors, and lesions in CT and MRI scans to measure their volume and plan surgery or radiotherapy.
• Autonomous driving: Perception systems segment road, sidewalk, vehicles, and pedestrians in every frame to find drivable space.
• Satellite image analysis: Land-cover maps label every pixel as water, forest, farmland, or buildings to track change over time.
• Photo and video editing: Background removal, portrait-mode blur, and sky replacement rely on person and sky masks computed on the device.
• Robotics and manufacturing: Robots use object masks to compute grasp points on irregular parts, and inspection systems measure the exact area of defects such as scratches and cracks.