Panoptic Segmentation

Panoptic segmentation is a computer vision task that assigns every pixel of an image two values: a class label and, for pixels that belong to a countable object, an instance ID. It combines semantic segmentation, which labels every pixel by class but merges objects of the same class, with instance segmentation, which separates individual objects but leaves the background unlabeled. The result is a single non-overlapping map of the whole scene in which road, sky, and grass are labeled as regions and each car and person is its own segment. Alexander Kirillov and colleagues at Facebook AI Research and Heidelberg University defined the task and its metric in the Panoptic Segmentation paper, published at CVPR 2019, choosing the word for its definition: including everything visible in one view.

Stuff and Things#

Panoptic segmentation divides classes into two kinds:

• Things: Countable objects with a defined shape, such as people, cars, animals, and handbags. Each one receives its own instance ID, so two people standing side by side become two segments.
• Stuff: Amorphous regions of similar texture or material, such as road, sky, grass, water, and walls. All pixels of a stuff class form one segment, because splitting a sky into instances has no meaning.

The COCO panoptic benchmark uses 133 classes, 80 things and 53 stuff, and the Cityscapes driving benchmark scores 19 classes, 8 things and 11 stuff. Pixels that are ambiguous or outside every class are marked void and ignored during evaluation. The COCO panoptic format stores each image as a PNG in which every segment has a unique ID, plus a JSON file mapping each ID to its category.

How Panoptic Models Work#

Early systems ran a semantic branch and an instance branch on a shared backbone and merged their outputs with fixed rules. Panoptic FPN, for example, resolves overlaps between instances by confidence, gives instances priority over stuff, and then removes stuff regions below an area threshold:

Bottom-up models such as Panoptic-DeepLab predict a semantic map together with each object's center and an offset from every pixel to its center, then group thing pixels by the center they point to. Universal models use one mask classification architecture for semantic, instance, and panoptic segmentation. Mask2Former predicts a fixed set of queries, each producing one mask and one class, which covers things and stuff alike and reached 57.8 PQ on COCO panoptic in its CVPR 2022 paper. The Hugging Face Transformers pipeline below runs a Mask2Former checkpoint trained on COCO panoptic:

import numpy as np
from transformers import pipeline

# Load Mask2Former trained on COCO panoptic: 80 thing and 53 stuff classes
segmenter = pipeline("image-segmentation", model="facebook/mask2former-swin-tiny-coco-panoptic")

# Each returned segment is one non-overlapping region with a class label and a binary mask
segments = segmenter("https://ultralytics.com/images/bus.jpg", subtask="panoptic")

for segment in segments:
    share = np.array(segment["mask"]).mean() / 255
    print(f"{segment['label']}: {share:.1%} of pixels")

The printout lists 10 segments. The four people, two handbags, and the bus (30.7% of the image) each appear as separate thing segments, next to one stuff segment each for pavement-merged (30.9%), building-other-merged (19.5%), and tree-merged (4.8%); the "-merged" suffix marks COCO stuff classes formed by combining finer labels. No pixel belongs to two segments, and the 0.4% of pixels that no segment claims are void.

Panoptic Quality (PQ)#

Panoptic predictions are scored with panoptic quality. A predicted segment matches a ground-truth segment of the same class when their Intersection over Union (IoU) exceeds 0.5; because segments do not overlap, each segment can have at most one match. Matches are true positives (TP), unmatched predictions are false positives (FP), and unmatched ground-truth segments are false negatives (FN). PQ is the sum of the IoUs of all matches divided by TP + ½FP + ½FN, which factors into two parts:

• Segmentation quality (SQ): The average IoU of the matched segments, measuring how well masks fit.
• Recognition quality (RQ): TP / (TP + ½FP + ½FN), the F1 score of segment detection.

PQ is computed per class and averaged, and benchmarks also report it separately for things and stuff.

Panoptic vs. Semantic and Instance Segmentation#

• Semantic segmentation: Answers "which class is this pixel?" for every pixel. Three adjacent cars become one car region.
• Instance segmentation: Answers "which object is this pixel part of?" for detected objects only. Background pixels get no label, and masks of different objects may overlap.
• Panoptic segmentation: Answers both questions for every pixel, with exactly one segment per pixel.

Ultralytics YOLO26 provides the two halves as separate tasks: -seg models return a mask and class for each object, and -sem semantic segmentation models return a class for every pixel, pretrained on the 19 Cityscapes classes or the 150 ADE20K classes.

Real-World Applications#

• Autonomous vehicles: One output gives the drivable road and sidewalks as stuff and every pedestrian and vehicle as a separate thing that a tracker can follow.
• Robotics: A mobile robot needs the floor it can drive on and each object it may pick up or avoid, in one consistent map.
• Medical image analysis: Digital pathology segments tissue regions as stuff and counts individual nuclei and cells as things.
• 3D and video: Video panoptic segmentation keeps instance IDs consistent across frames, and LiDAR panoptic segmentation, benchmarked on SemanticKITTI and nuScenes, labels every 3D point of a scan with a class and an instance ID.