Letterboxing

Letterboxing is an image-resizing technique that preserves an image's aspect ratio while adding padding to reach a target size. It scales the image uniformly, then fills unused space with border pixels, keeping objects' proportions and the full field of view. In computer vision, it prepares differently shaped images for models that accept a common input shape.

Outside AI, letterboxing also describes displaying widescreen film with horizontal bars.

How Letterboxing Preserves Image Geometry#

Aspect ratio is the relationship between image width and height. During data preprocessing, letterboxing selects the smaller of two scale factors: target width divided by original width, and target height divided by original height. Both dimensions use that factor, so the resized image fits inside the target. Remaining pixels become padding. TensorFlow's resize-and-pad operation implements this approach with zero-valued borders.

For example, a 1280 x 720 image fitted into a 640 x 640 target uses a scale factor of 0.5. Its content becomes 640 x 360, leaving 280 rows of padding: 140 above and 140 below when centered. Portrait images receive side borders. A letterboxed image can therefore have horizontal or vertical padding; the target can also be rectangular.

Ultralytics YOLO uses the LetterBox transform to preserve geometry for detection, segmentation, and pose inputs, with gray padding valued at 114 by default. For YOLO26 detection, segmentation, and pose prediction, minimum-rectangle padding reduces borders to a compatible stride multiple when batch shapes and the inference backend support it.

Letterboxing vs. Stretching and Cropping#

Three resizing choices handle mismatched image and target shapes differently:

• Letterboxing preserves proportions and the full image, allocating part of the input to padding.
• Stretching resizes width and height independently to fill the target. Circles become ellipses when the scale factors differ.
• Cropping preserves proportions when paired with uniform resizing, then removes content outside the selected region. Objects near image edges can be cut off.

These choices affect object detection, where shapes and positions matter, and image classification, where cropping changes the visible evidence for a class.

Padding also consumes input area. In the 1280 x 720 example, image content occupies 56.25% of the square and padding occupies 43.75%. Downsampling reduces fine detail inside that content; adding border pixels creates no additional object detail. For high-resolution images containing small objects, tiled inference processes separate image regions at a larger effective object scale.

A Python Letterboxing Example#

Install Pillow with pip install pillow. Its ImageOps.pad operation resizes and centers an image on a padded canvas. This example uses the bus photograph distributed for vision tutorials.

from urllib.request import urlretrieve

from PIL import Image, ImageOps

urlretrieve("https://ultralytics.com/images/bus.jpg", "bus.jpg")
image = Image.open("bus.jpg")
image = image.convert("RGB")

# Preserve proportions and fill the unused area with gray.
letterboxed = ImageOps.pad(image, (640, 640), color=(114, 114, 114))
letterboxed.save("letterboxed_bus.png")
print(f"{image.size} -> {letterboxed.size}")

The output dimensions are (810, 1080) -> (640, 640), expressed as width and height. The saved image contains the complete photograph resized to 480 x 640, with an 80-pixel border on each side.

Mapping Annotations and Predictions#

Letterboxing transforms the coordinate system as well as the pixels. Every bounding box coordinate must be scaled and shifted by the left or top padding. The same geometry applies to keypoints in pose estimation and object outlines in instance segmentation. Leaving annotations unchanged during training misaligns targets with the objects they describe.

After inference, coordinate rescaling reverses the transformation: subtract the padding offset, then divide by the resize scale. For an otherwise correctly positioned box with nonzero padding, omitting that offset shifts the box and lowers its intersection over union with the ground truth. In the bus example, a point at padded coordinates (320, 320) maps to original coordinates (405, 540) after subtracting 80 from x and dividing both coordinates by 640 / 1080.