Semantic vs Instance Segmentation: How to Choose the Right Task
Semantic segmentation labels every pixel by class; instance segmentation separates each object. Learn which one your computer vision project needs, and why.
Semantic segmentation and instance segmentation both label images at the pixel level, but they answer different questions. Semantic segmentation assigns a class to every pixel, so all the cars in a scene become one "car" region. Instance segmentation separates each object, so every car gets its own mask. Choosing between them comes down to one question: do you need to know what is at each pixel, or which object each pixel belongs to? This guide helps you make that choice. It also covers what each task costs to label and how each is evaluated. Ultralytics YOLO supports both tasks in the same framework.
The difference in one line#
- Semantic segmentation classifies every pixel into a category, such as road, sky, person or car. Objects of the same class merge into one region. See the semantic segmentation glossary entry.
- Instance segmentation finds each individual object and gives it its own mask and ID. It usually covers countable objects and ignores background such as sky or grass. See the instance segmentation glossary entry.
Fig 1. The same street with both tasks. Ultralytics YOLO26x-sem (Cityscapes classes) labels every pixel, so all cars share one "car" class. Ultralytics YOLO26x-seg gives each of the 8 cars and 3 people it found its own mask (confidence 0.5 or higher). The photo was generated for this article; the masks are real model outputs.
Which task does your project need?#
Start from the question your application has to answer:
| If your application needs to… | Use | Example |
|---|---|---|
| Know how much of the scene is a given class | Semantic | Drivable road area, crop vs weed coverage, land cover |
| Label background regions as carefully as objects | Semantic | Sky, water, vegetation, tissue types in a scan |
| Count objects | Instance | Fruit on a tree, people in a queue |
| Track objects across video frames | Instance, plus a tracker | People or vehicles moving through a scene |
| Measure or act on one object at a time | Instance | A robot picking one part from a pile, sizing each defect |
| Do both: full scene coverage and separate objects | Panoptic | Street scenes for autonomous driving |
If you're unsure, ask whether two touching objects of the same class must be told apart. If yes, you need instance segmentation (or panoptic). If no, semantic segmentation is simpler and usually cheaper to run.
Fig 2. Seven shoppers standing close together in a checkout queue. In the semantic output (Ultralytics YOLO26x-sem) they merge into two connected "person" regions, so you can't count them from it. The instance output (Ultralytics YOLO26x-seg) separates all 7. The photo was generated for this article; the masks are real model outputs.
Key differences at a glance#
| Criteria | Semantic segmentation | Instance segmentation |
|---|---|---|
| Question answered | What class is each pixel? | Which object does each pixel belong to? |
| Objects of the same class | Merged into one region | Separated, each with its own mask and ID |
| Coverage | Every pixel, including background | Usually only the objects of interest |
| Output | One class map the size of the image | A list of masks, each with a class and confidence |
| Can you count objects? | Not directly | Yes |
| Main metric | mean Intersection over Union (mIoU) | Mask mean Average Precision (mask mAP) |
| Relative complexity | Lower: per-pixel classification | Higher: detection plus per-object masks |
What each task costs to label#
The labeling effort is often what decides the choice in practice.
- Semantic datasets assign a class to every pixel of every image, background included. They're usually stored as single-channel mask images. Every image is fully labeled, so no pixel can be skipped unless it's explicitly marked "ignore".
- Instance datasets outline each object of interest separately, as a polygon or mask. Background can be left unlabeled, but every object needs its own outline, and crowded scenes multiply the work.
Both are labor-intensive. Assisted annotation, where a click or a box becomes a mask, cuts the effort for either task. If you only need region-level answers, don't pay for instance labels.
How each task is evaluated#
- Semantic segmentation is measured mainly with mIoU: the overlap between predicted and true regions, averaged across classes. Pixel accuracy is also reported. IoU is the same overlap measure used in detection.
- Instance segmentation is measured with mask mAP, the mean Average Precision used for detection but computed on mask overlap. It rewards finding each object, not just covering the right pixels.
These metrics aren't comparable with each other. A high mIoU says nothing about whether individual objects were separated.
Panoptic segmentation: when you need both#
Panoptic segmentation gives every pixel a class and, for countable objects, an instance ID. Background "stuff" such as road and sky is labeled semantically. Foreground "things" such as cars and people are separated one by one. It's measured with Panoptic Quality (PQ). Choose it when you need full scene coverage and per-object separation at once. See panoptic segmentation.
How each works, briefly#
Semantic segmentation models are typically encoder-decoder networks that predict a class for every pixel. U-Net, DeepLab and SegFormer are well-known examples. Instance segmentation models usually build on object detection. Mask R-CNN predicts a mask inside each detected box, while single-stage models such as YOLACT and the Ultralytics YOLO segmentation models predict boxes and masks in one pass for real-time use. Universal architectures such as Mask2Former handle semantic, instance and panoptic segmentation in one model. For more on instance segmentation, see what is instance segmentation.
Both tasks in Ultralytics YOLO#
Ultralytics YOLO runs both tasks with the same Python API and CLI:
- Instance segmentation uses the
segmenttask. - Semantic segmentation uses the
semantictask. YOLO26 semantic models carry a-semsuffix and come pretrained on Cityscapes and ADE20K. The task page includes a table showing exactly how the two tasks' outputs differ in code.
Both can be trained locally or in the cloud on Ultralytics Platform.
Fig 3. Semantic segmentation answers "how much of the scene is each class". Ultralytics YOLO26x-sem labels 51.3% of this view as sky, 18.2% as vegetation, 15.2% as road and 14.9% as terrain. The photo was generated for this article; the class map and shares are real model outputs.
Frequently asked questions
Semantic segmentation labels every pixel with a class but treats all objects of the same class as one region. Instance segmentation separates each object into its own mask, so objects of the same class can be counted and tracked individually.
Ask whether two touching objects of the same class must be told apart. If you need to count, track or act on individual objects, use instance segmentation. If you only need to know which regions belong to which class, semantic segmentation is simpler.
Mask R-CNN is an instance segmentation model. It detects each object, then predicts a separate pixel mask inside that object's bounding box.
Instance segmentation is generally harder. It has to classify pixels and also split objects of the same class apart, which requires detection as well as segmentation.
Mostly with mean Intersection over Union (mIoU), the average overlap between predicted and true regions across classes. Instance segmentation uses mask mAP, and panoptic segmentation uses Panoptic Quality (PQ).
