Interactive Segmentation

Interactive segmentation produces a pixel-accurate mask of an object from a few hints given by a person, such as clicks, a bounding box, or scribbles, and refines the mask as the person adds corrections. A positive click marks a point that belongs to the object and a negative click marks one that does not, so after each round the user only has to point at the largest remaining error. The approach is class-agnostic: the model segments whatever the user indicates, including objects it never saw during training, which makes it a standard tool for data annotation, photo editing, and delineating structures in medical scans. It is a human-in-the-loop form of image segmentation.

How the Click Loop Works#

The model receives the image together with every click so far, and many models also receive the previous mask, then predicts a new mask. The loop repeats until the mask is good enough:

Benchmarks simulate this user: each new click goes to the center of the largest error region, and the score is the number of clicks needed to reach an intersection over union of 85% or 90% with the ground truth, reported as NoC@85 and NoC@90 on datasets such as GrabCut, Berkeley, and DAVIS.

The Ultralytics package runs Meta's SAM models with point prompts, where a label of 1 marks a positive click and 0 a negative one:

from ultralytics import SAM

# Meta's SAM 2.1 base model, run through the Ultralytics package
model = SAM("sam2.1_b.pt")
image = "https://ultralytics.com/images/bus.jpg"

# Round 1: a single positive click on the pedestrian's dark coat
results = model(image, points=[[[250, 620]]], labels=[[1]])
print(f"1 click: {int(results[0].masks.data.sum())} mask pixels")

# Round 2: add a positive click on the pedestrian's jeans and a negative click on the bus
results = model(image, points=[[[250, 620], [290, 760], [420, 560]]], labels=[[1, 1, 0]])
print(f"3 clicks: {int(results[0].masks.data.sum())} mask pixels")
results[0].save(filename="interactive_mask.jpg")

It prints 32,598 mask pixels after one click and 33,120 after three, so the single click on the coat already selects nearly the whole pedestrian and the extra clicks refine the mask by about 500 pixels. Ultralytics Platform runs the same loop in its annotation editor: in Smart mode, a click outside the current SAM mask adds that region and a click inside subtracts it, and the finished mask is saved as a polygon, bounding box, or oriented box for the dataset's task.

From Graph Cuts to Promptable Models#

Early tools optimized boundaries directly. Intelligent Scissors (1995) snapped a traced outline to strong edges, and GrabCut, published by Microsoft Research in 2004 and included in OpenCV, segments the object inside a user-drawn box by iteratively fitting color models to foreground and background and solving a graph cut. Deep learning methods since 2016 encode clicks as extra input channels next to the image and learn the mask with a convolutional neural network, and later models also take the previous mask as input to refine it.

Meta's Segment Anything Model (SAM), released in 2023, split the work so interaction is nearly instant: a heavy vision transformer encodes the image once, and a light prompt encoder and mask decoder turn each new set of clicks into a mask in about 50 milliseconds. SAM was also trained with its own interactive loop: while building the SA-1B dataset, annotators clicked to segment objects with successive versions of the model, and the average time per mask fell from 34 to 14 seconds. SAM 2 extends the clicks to video, propagating a mask through the frames, and SAM 3 adds promptable concept segmentation, finding every instance that matches a short text phrase.

Interactive vs. Automatic Segmentation#

Instance segmentation models segment a fixed set of trained classes automatically, with no human input, which suits production pipelines that must process every frame. Interactive segmentation trades that automation for flexibility and is how the training masks for such models are often created, frequently inside an active learning loop that sends the model's least certain images to annotators. Visual prompting is the broader idea of steering a vision model with points, boxes, or example images, and interactive segmentation is its iterative, mask-producing form.

Real-World Applications#

• Dataset labeling: Annotators outline objects with a few clicks instead of tracing hundreds of polygon vertices, and SAM-assisted annotators needed 14 seconds per mask, 6.5 times faster than the manual mask annotation of COCO.
• Medical imaging: Radiologists delineate tumors and organs in CT and MRI scans with boxes and corrective clicks, and MedSAM adapted SAM to medicine by fine-tuning it on 1,570,263 medical image-mask pairs covering 10 imaging modalities.