Image Inpainting

Image inpainting is the task of filling a masked region of an image, such as a scratch, a removed object, or a missing patch, with content that blends into its surroundings. Every method takes two inputs, the image and a binary mask marking the pixels to replace, and keeps everything outside the mask unchanged. The term comes from art conservation, where restorers retouch damaged paintings, and Marcelo Bertalmío, Guillermo Sapiro, Vicent Caselles, and Coloma Ballester brought it to digital computer vision at SIGGRAPH 2000 with a method that extends lines of equal brightness from the hole's border into the hole. The pixels under the mask are unknown, so an inpainted region is always a plausible invention, never a recovery of what was there.

Classical, Patch-Based, and Learned Methods#

Classical methods propagate color and edges inward from the mask boundary. OpenCV ships two of them in its inpainting module: Alexandru Telea's 2004 fast marching method (cv2.INPAINT_TELEA) and the 2001 Navier-Stokes method of Bertalmío, Andrea Bertozzi, and Sapiro (cv2.INPAINT_NS). They work well on dust, scratches, and thin text overlays and blur into a smear on large holes. The snippet below gives Telea's method the same number of missing pixels twice, once as thin scratches and once as a single square:

import cv2
import numpy as np
from skimage import data

image = cv2.cvtColor(data.astronaut(), cv2.COLOR_RGB2BGR)  # 512x512 sample photo

# Two masks with about the same pixel count: thin scratches and one solid block
scratches = np.zeros(image.shape[:2], np.uint8)
for y in range(40, 512, 40):
    cv2.line(scratches, (0, y), (511, y + 30), 255, thickness=2)
block = np.zeros_like(scratches)
side = int(np.sqrt(np.count_nonzero(scratches)))
block[200 : 200 + side, 200 : 200 + side] = 255

for name, mask in [("scratches", scratches), ("block", block)]:
    restored = cv2.inpaint(image, mask, 3, cv2.INPAINT_TELEA)
    # Mean error inside the mask, where every pixel was reconstructed
    error = np.abs(restored.astype(int) - image.astype(int))[mask > 0].mean()
    print(f"{name}: {np.count_nonzero(mask)} masked pixels, mean error {error:.1f} gray levels")

It prints a mean error of 9.1 gray levels for 18,432 scratch pixels and 36.5 for an 18,225-pixel block. Every scratch pixel has known neighbors a pixel or two away, while the center of the block is 67 pixels from the nearest real one, so the same missing area costs four times the error.

Patch-based methods copy texture from elsewhere in the image instead of smoothing it in. Antonio Criminisi, Patrick Pérez, and Kentaro Toyama's 2004 exemplar-based algorithm fills a hole patch by patch, starting where strong edges enter it, and the PatchMatch nearest-patch search of Connelly Barnes and colleagues (2009) made the approach fast enough for interactive use; it underlies Photoshop's Content-Aware Fill. Patch methods excel at grass, brick, and sky and fail when the missing content appears nowhere else in the picture.

Learned models infer content from what they saw in training. Deepak Pathak and colleagues' Context Encoders (CVPR 2016) trained a convolutional neural network with an adversarial loss, borrowed from generative adversarial networks, to complete a masked center region. Samsung AI's LaMa (WACV 2022) uses fast Fourier convolutions, whose receptive field spans the whole image from the first layer, and trains on large random masks, so it completes big holes and repeating structures such as fences and windows. Diffusion models fill a hole by denoising it conditioned on the visible pixels and, optionally, a prompt: Stable Diffusion inpainting checkpoints give the U-Net five extra input channels for the mask and the masked image's latent, and Andreas Lugmayr and colleagues' RePaint (CVPR 2022) uses an unmodified diffusion model, replacing the known region with a noised copy of the original at every step. Adobe's Generative Fill brought prompt-driven text-to-image inpainting into Photoshop in 2023.

Masks and Object Removal#

Object removal is image segmentation followed by inpainting: a mask selects the object, and inpainting fills the hole it leaves. Masks traced tightly around an object leave its edge pixels, shadow, and reflection behind as a ghost, so editors dilate the mask by a few pixels and extend it over the shadow. Instance segmentation produces one mask per object, which makes it the usual source of removal masks, and promptable models such as Meta's SAM 2 turn a click into a mask for objects outside any class list. YOLO26 segmentation models return each object's outline as a polygon in result.masks.xy, which cv2.drawContours with cv2.FILLED rasterizes into an inpainting mask, the same step the isolating segmentation objects guide uses to cut objects out.

Video inpainting adds time: the fill must match across frames or it flickers, so AI video editing tools propagate masks with tracking and borrow background pixels revealed in neighboring frames as the camera moves.

Related Tasks#

• Outpainting: Uses the same models to extend an image past its original borders, with the new canvas area as the mask.
• Super-resolution: Adds detail across the whole image at a higher resolution, with no mask.
• Denoising: Removes noise spread over every pixel while keeping the scene, again with no mask.
• Frame interpolation: Synthesizes entire in-between frames of a video from the frames on either side.

Applications and Provenance#

Photo restoration masks cracks, dust, and tears in scanned prints, and film restoration does the same frame by frame for scratches on archival footage. Retouching and product photography remove stray objects, power lines, and blemishes. In satellite image analysis, gap filling is a classic inpainting problem: the scan line corrector on Landsat 7 failed on May 31, 2003, leaving wedge-shaped stripes that cover about 22 percent of every later scene.

Learned inpainting is evaluated on held-out images with synthetic masks, using Fréchet Inception Distance (FID) for realism, since a good fill can differ from the original pixels and still be correct. The same realism makes inpainting a tool for deepfakes and for removing watermarks, and in medical, scientific, or forensic images an invented structure can change a diagnosis or a verdict. Content Credentials record such edits in signed metadata, and Adobe attaches them to images edited with Generative Fill.