Visual Prompting

Visual prompting is the practice of telling a computer vision model what to segment or detect by marking the image itself, with a click, a box, a scribble, a mask, or an example, instead of describing the target in words. Promptable foundation models such as Meta's Segment Anything Model (SAM) turn each mark into an embedding and predict a mask for the indicated object without any retraining. A visual prompt points at one specific instance or shows an example of what to find, which helps when the target is hard to name in a text prompt, such as a particular defect or an unfamiliar part. Visual prompt tuning shares the name but is a prompt tuning method that learns extra input tokens during fine-tuning, with no one marking the image.

How Visual Prompting Works#

SAM, released by Meta in 2023 with the SA-1B dataset of 1.1 billion masks on 11 million images, splits the work into three parts. A large image encoder, a vision transformer, runs once per image; a lightweight prompt encoder turns points and boxes into positional embeddings; and a mask decoder combines the two to predict a mask in about 50 milliseconds in a web browser, so each new click updates the result interactively. A single click is ambiguous, since it could mean a shirt or the person wearing it, so SAM can return three candidate masks with a quality score for each. Meta's SAM 2 (2024) extends prompts to video, carrying a clicked object through the following frames with a memory of earlier ones. SAM 2.1 runs in the Ultralytics package, where a click is a point with a label of 1 for foreground or 0 for background:

from ultralytics import SAM

# Load Meta's SAM 2.1 base model
model = SAM("sam2.1_b.pt")

# One foreground click (label 1) on the man in the dark coat
results = model("https://ultralytics.com/images/bus.jpg", points=[280, 600], labels=[1])
result = results[0]

# Save the image with the mask predicted for the clicked object
result.save(filename="sam_point_prompt.jpg")

The saved image shows the mask SAM 2.1 predicts from that single click. Interactive segmentation repeats this step until the mask is right, and the smart annotation mode of Ultralytics Platform works this way: each click becomes a SAM point prompt, a click outside the current mask adds that region, a click inside removes it, and the result is saved as a polygon, a box, or an oriented box depending on the dataset's task.

Types of Visual Prompts#

• Points and boxes: Foreground and background clicks select or exclude regions, and a rough box around an object is less ambiguous than a click, which also lets a detector's boxes prompt a segmenter automatically.
• Masks and scribbles: A coarse mask from a previous step, or strokes drawn over the object, refine a prediction that clicks alone get wrong.
• Exemplar boxes: A box around one example asks the model for every object like it, so counting all the screws of one type takes one example box. Open-set detectors such as T-Rex2 and YOLOE, and Meta's SAM 3 (2025), accept these exemplar prompts.
• In-context examples: An example image paired with its desired output, such as a mask or an edge map, defines the task itself, and the model completes the output for a new image by analogy.
• Marks for vision language models: Set-of-Mark prompting draws numbered marks over image regions so the model can refer to "region 3" in its answer, which improves its visual grounding.

Real-World Applications#

• Data annotation: A few clicks per object replace tracing a polygon point by point, which cuts the cost of building instance segmentation datasets.
• Medical imaging: MedSAM adapted SAM to 1,570,263 medical image-mask pairs across 10 imaging modalities, so a clinician's box around an organ or lesion returns its outline.
• Robotics: An operator clicks an object in a robot's camera view, and the resulting mask gives the robot the object's exact pixels for grasp planning.