Image Inpainting

Image inpainting is a computer vision technique that reconstructs missing, damaged, or intentionally masked regions of an image so the completed area blends with its surroundings. Instead of simply covering unwanted pixels, an inpainting system estimates plausible replacement content from nearby texture, scene structure, and, in generative systems, learned visual patterns. Common goals include restoring damaged photographs, removing objects or text, and filling gaps created during image editing.

How Image Inpainting Works#

An inpainting workflow usually has two inputs: the original image and a mask identifying the pixels to replace. The system preserves pixels outside the mask while synthesizing or reconstructing content inside it. Mask conventions vary by tool, so developers must confirm whether white, black, zero, or nonzero pixels represent the editable region.

Traditional computer vision methods propagate colors, edges, and textures inward from the mask boundary. For example, the OpenCV image inpainting workflow fills masked areas using neighboring pixels and offers algorithms designed to continue surrounding structures. These approaches are efficient and often work well for scratches, dust, thin text, and small holes. (docs.opencv.org)

Learning-based systems infer higher-level context. A model might recognize that a masked area lies on brickwork, grass, clothing, or a person’s face and generate content consistent with that interpretation. However, the output is plausible rather than a recovery of the unknown original pixels.

Masks and Model Approaches#

Mask quality strongly affects the result. A mask that is too small may leave object edges or shadows behind, while an oversized mask removes useful context. Precise masks can be created manually or generated with image segmentation. Instance segmentation is especially useful because it produces a separate pixel-level mask for each object.

Ultralytics SAM 2 documentation describes point- and box-prompted segmentation for interactively selecting an object. For known object classes, the Ultralytics YOLO segmentation task returns object masks and polygon coordinates. Teams can also create and refine polygon masks with the Ultralytics Platform annotation editor, which supports manual, SAM-assisted, and YOLO-assisted annotation.

Several model families can perform the actual filling:

• Classical methods extend nearby pixels, contours, or smooth intensity patterns. The scikit-image restoration API includes biharmonic inpainting for masked regions.
• Generative adversarial networks learn to produce visually realistic completions through competition between generator and discriminator networks.
• Diffusion models iteratively transform noise inside the mask into content conditioned on the visible image and, optionally, a text prompt.

Image Inpainting vs Related Tasks#

Inpainting is defined by localized replacement. Related tasks have different objectives:

• Object removal is an application of inpainting: segmentation selects the object, and inpainting fills the resulting hole.
• Image segmentation identifies which pixels belong to an object or class but does not replace them.
• Denoising reduces widespread noise while generally preserving scene content.
• Outpainting extends content beyond the original canvas rather than repairing an internal region. The Google Cloud mask-based image editing guide distinguishes inserting, removing, expanding, and replacing content.
• Super-resolution increases image resolution and detail instead of targeting a user-defined mask.

Removing text follows the same workflow as object removal. The text is masked, then the system predicts the obscured background. If the text covers a unique face, symbol, or document detail, the original information cannot be reliably recovered.

Real-World Applications#

Retail image cleanup: An online seller can segment a stray cable, reflection, support stand, or background object from a product photograph and inpaint the area. Small defects may suit Adobe Content-Aware Fill, while Adobe generative image editing can handle larger, semantically complex regions. (helpx.adobe.com)

Image restoration: Archives and imaging teams can mask scratches, dust spots, sensor defects, or missing pixels in scanned photographs and microscopy frames. This can improve readability and presentation, but reconstructed areas should not be treated as observed evidence. In medical, scientific, or forensic images, an invented structure could change interpretation and must be clearly documented.

Practical Workflow and Limitations#

The following example uses YOLO26 to create a person mask and OpenCV to fill that region. Install the dependencies with pip install ultralytics opencv-python numpy.

import cv2
import numpy as np
from ultralytics import YOLO

model = YOLO("yolo26n-seg.pt")
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Select the first detected person and convert its polygon to a mask.
class_ids = result.boxes.cls.cpu().numpy().astype(int)
person_index = next(i for i, class_id in enumerate(class_ids) if class_id == 0)
contour = result.masks.xy[person_index].astype(np.int32)
contour = contour.reshape(-1, 1, 2)

image = result.orig_img.copy()
mask = np.zeros(image.shape[:2], dtype=np.uint8)
cv2.drawContours(mask, [contour], -1, 255, cv2.FILLED)

inpainted = cv2.inpaint(image, mask, 3, cv2.INPAINT_TELEA)
cv2.imwrite("inpainted.jpg", inpainted)

This combines object localization with the documented pattern for building masks from segmentation contours. Classical filling may struggle with large objects, repeating geometry, shadows, reflections, or faces. Generative methods handle broader context but can introduce fabricated details. Preserve the original, inspect boundaries at full resolution, and record significant edits using provenance mechanisms such as C2PA Content Credentials, especially when authenticity matters. NIST guidance on synthetic-content transparency also emphasizes provenance and labeling as important safeguards for meaningfully altered media. (c2pa.org)