Image Matting

Image matting is the process of estimating how much foreground contributes to each pixel in an image. Its output, an alpha matte, preserves partially transparent details such as individual hairs, fur, and soft edges so a subject can be placed naturally against another background. Also called alpha matting, this computer vision task supports precise cutouts and image compositing. Physical picture-framing mats and medical floor matting are separate meanings of "matting."

How an Alpha Matte Works#

An alpha matte is a single-channel image with values between 0 and 1: 0 represents background, 1 represents fully opaque foreground, and intermediate values represent partial foreground coverage or transparency. Alpha describes the pixel's mixture; a model's confidence describes certainty about a prediction. Fine hair can occupy only part of a pixel, producing an intermediate alpha value even when the strand itself is opaque.

The standard compositing relationship is image = alpha * foreground + (1 - alpha) * background. Matting estimates these components from an existing image; compositing combines the extracted foreground with a replacement background. Estimating foreground color also matters: retaining the original background's color around hair creates colored fringes in the new composite.

Image segmentation assigns pixels to regions. Semantic segmentation assigns class labels, while instance segmentation separates individual objects. Matting estimates continuous foreground opacity. Ultralytics YOLO26 segmentation supports object isolation by turning instance masks into transparent cutouts; those masks can provide an initial foreground outline for a separate matting refinement stage.

Trimaps and Matting Methods#

A trimap supplies three regions: known foreground, known background, and an unknown region containing boundaries or transparent material. This form of data annotation constrains the estimation problem. Like interactive segmentation, it lets users guide extraction through a small amount of input. Incorrectly marking hair as definite background forces its alpha to zero, removing it from the cutout.

Closed-form matting, introduced by Anat Levin, Dani Lischinski, and Yair Weiss in 2006 through A Closed Form Solution to Natural Image Matting, uses local foreground and background color assumptions to construct a sparse linear system. Solving that system estimates alpha throughout the unknown region.

Deep learning methods learn appearance and boundary patterns from labeled examples. Ning Xu, Brian Price, Scott Cohen, and Thomas Huang introduced Deep Image Matting in 2017, combining a convolutional neural network encoder-decoder with refinement. Its Adobe matting data and Composition-1k test set became established evaluation resources. Compositing foregrounds onto varied backgrounds expands training data through data augmentation.

Estimating a Matte in Python#

The PyMatting alpha-estimation API implements closed-form matting. Install it with pip install pymatting pillow, then use foreground and background seed regions on the sample image. Here, gray trimap values leave pixels for the solver to estimate.

from urllib.request import urlopen

import numpy as np
from PIL import Image
from pymatting import estimate_alpha_cf

with urlopen("https://ultralytics.com/images/bus.jpg") as response:
    photo = Image.open(response).convert("RGB")
photo.thumbnail((256, 256))
image = np.asarray(photo, dtype=np.float64) / 255.0

# Mark sky as background and a patch of the bus as foreground.
trimap = np.full(image.shape[:2], 0.5)
trimap[:20, :] = 0.0
trimap[120:145, 115:145] = 1.0

alpha = estimate_alpha_cf(image, trimap)
matte = Image.fromarray(np.uint8(np.clip(alpha, 0, 1) * 255))
matte.save("alpha_matte.png")

The saved grayscale image shows the estimated foreground contribution: white at foreground seeds, black at background seeds, and estimated transitions elsewhere.

Applications and Quality Evaluation#

Portrait editing, product photography, and film compositing use mattes to preserve fine boundaries during background replacement. Portrait video matting extends this operation across frames, where unstable boundaries produce visible flicker. MODNet provides a verifiable example of trimap-free portrait extraction from RGB input for real-time inference. Image inpainting instead synthesizes content inside a selected region.

Matting evaluation compares predicted alpha values with reference mattes. The perceptually motivated matting benchmark, introduced by Christoph Rhemann and colleagues in 2009, uses sum of absolute differences, mean squared error, gradient error, and connectivity error. These complement region-overlap measures such as Intersection over Union: gradient error measures discrepancies in alpha transitions, while connectivity error measures disruptions to connected foreground structure.