Instance Segmentation

Instance segmentation is a computer vision task that finds each object in an image, assigns it a class, and outlines it with its own pixel-level mask. It combines the per-object reasoning of object detection with the pixel precision of image segmentation: where a detector draws a rectangular bounding box that also contains background, an instance segmentation model marks exactly which pixels belong to which object. Two people standing shoulder to shoulder become two separate masks with two labels, which is what makes it possible to count, measure, and cut out individual objects.

Instance vs. Semantic and Panoptic Segmentation#

Instance segmentation is one of three segmentation tasks that differ in what a label means:

• Semantic segmentation: Labels every pixel with a class and merges all objects of that class into one region, so three parked cars become one "car" area. It also labels background regions such as road and sky.
• Instance segmentation: Gives each countable object, or "thing," its own mask and ID, so the three cars become car 1, car 2, and car 3. Pixels that belong to no detected object stay unlabeled.
• Panoptic segmentation: Assigns every pixel a class and also an instance ID for countable objects, joining the two views in a single output.

Promptable models such as Meta's Segment Anything Model (SAM) perform interactive segmentation: they return a mask for whatever a click, box, or text prompt points to, without a fixed list of classes.

How Instance Segmentation Models Work#

Two-stage models detect first and segment second. Mask R-CNN, the long-standing reference design from 2017, proposes candidate regions, classifies and refines each box, and predicts a small 28×28 mask inside every box with a separate mask branch.

One-stage models predict boxes and masks in a single pass, which suits real-time inference. YOLACT introduced the prototype design that YOLO segmentation models also use: the network predicts a shared set of prototype masks for the whole image, plus a short vector of mask coefficients for each detected object. Each object's mask is the coefficient-weighted sum of the prototypes, cropped to its box and thresholded into a binary mask.

Transformer-based models such as Mask2Former take a third route: they predict a fixed set of mask queries and use the same architecture for semantic, instance, and panoptic segmentation.

Masks, Polygons, and Metrics#

Training labels are usually polygons traced around each object, stored in formats such as COCO JSON or YOLO text files with one class ID and a list of normalized vertices per object. At inference, models output each instance as a binary mask, which can be converted back into a polygon outline for storage or drawing. Accuracy is reported as mask mean average precision (mAP), computed like box mAP except that Intersection over Union (IoU) is measured between predicted and ground-truth masks. Tracing a polygon around every object takes far longer than drawing a box, so promptable models such as SAM are widely used to turn a single click into a draft mask that an annotator then corrects.

Ultralytics YOLO26 segmentation models, marked by the -seg suffix, use the prototype design with 32 prototypes and 32 coefficients per detection and return each instance with its class, box, mask, and polygon outline:

from ultralytics import YOLO

# Load a YOLO26 instance segmentation model pretrained on the 80 COCO classes
model = YOLO("yolo26n-seg.pt")

# Segment every object instance in the image
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Each instance has its own class and mask outline, even when classes repeat
for class_id, polygon in zip(result.boxes.cls.int().tolist(), result.masks.xy):
    print(f"{result.names[class_id]}: outline with {len(polygon)} points")

result.save(filename="instance_segmentation.jpg")

Every detected person prints as a separate line with its own outline, and the saved image colors each instance's mask separately. To label your own data and train a model, see how to custom-train YOLO26 for instance segmentation or the Ultralytics Academy lesson Label and Train Instance Segmentation.

Real-World Applications#

Instance masks matter wherever the exact shape or count of individual objects drives a decision:

• Robotic picking: A robot arm grasping parts from a cluttered bin needs each part's exact outline to pick a grasp point clear of its neighbors.
• Cell biology: Microscopy pipelines segment individual cells and nuclei, including touching ones, to count them and measure their size and shape.
• Agriculture: Harvesting robots and yield estimators separate individual fruits in dense canopies to count them and estimate their size from mask area.
• Video object tracking: Combining instance masks with a tracker follows each object's exact silhouette from frame to frame.
• Autonomous vehicles: Driving datasets such as Cityscapes provide instance-level labels for eight classes of people and vehicles, so perception models can separate each pedestrian in a crowd.