Image Registration

Image registration is the computer vision process of aligning two or more images of the same scene so that each physical point lands at the same pixel coordinates in all of them. The images may come from different times, viewpoints, or sensors: a satellite pass in January and another in June, an MRI and a CT scan of the same patient, or consecutive frames from a shaking camera. Once registered, the images can be compared pixel by pixel, fused into one view, or stitched into a mosaic.

Registration designates one image as the fixed, or reference, image and transforms the other, the moving image, into its coordinate system. That involves three choices: how to measure whether the images line up, what kind of geometric transformation is allowed, and how to search for the best one. After the transformation is estimated, the moving image is resampled, with interpolation computing pixel values at the new positions.

Feature-Based and Intensity-Based Methods#

Feature-based registration detects distinctive points, such as corners and blobs, in both images, describes each with a descriptor such as ORB or SIFT, and matches descriptors across images. The matched keypoints give pairs of corresponding locations, and a robust estimator such as RANSAC fits a transformation while rejecting mismatches. Feature methods are fast and handle large displacements, which makes them the usual choice for panorama stitching and drone mapping, and learned image matching extends them to scenes where hand-crafted descriptors struggle. This example warps a test photo by a known rotation, scale, and shift, then recovers the transformation with OpenCV:

import cv2
import numpy as np
from skimage import data

# Fixed image, and a moving copy rotated 10 degrees, scaled 1.1x, and shifted (25, -15) pixels
fixed = data.camera()
warp = cv2.getRotationMatrix2D((256, 256), 10, 1.1)
warp[:, 2] += (25, -15)
moving = cv2.warpAffine(fixed, warp, (512, 512))

# Detect and match ORB features in both images
orb = cv2.ORB_create(1000)
kp_moving, des_moving = orb.detectAndCompute(moving, None)
kp_fixed, des_fixed = orb.detectAndCompute(fixed, None)
matches = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True).match(des_moving, des_fixed)
src = np.float32([kp_moving[m.queryIdx].pt for m in matches])
dst = np.float32([kp_fixed[m.trainIdx].pt for m in matches])

# Fit a similarity transform (rotation, scale, translation) with RANSAC, rejecting wrong matches
M, inliers = cv2.estimateAffinePartial2D(src, dst, method=cv2.RANSAC)
angle, scale = np.degrees(np.arctan2(M[1, 0], M[0, 0])), np.hypot(M[0, 0], M[1, 0])
print(f"Kept {inliers.sum()} of {len(matches)} matches: rotation {angle:.2f} degrees, scale {scale:.3f}")

# Resample the moving image into the fixed image's coordinates
registered = cv2.warpAffine(moving, M, (512, 512))
cv2.imwrite("registered.png", registered)

It prints Kept 497 of 567 matches: rotation 10.01 degrees, scale 0.910. The estimated transform is the inverse of the applied warp, a 10-degree rotation back and a scale of 1/1.1, and registered.png lines up with the original photo wherever the two overlap.

Intensity-based registration skips explicit features and compares pixel values directly, adjusting the transformation to maximize a similarity measure. The idea goes back to Bruce Lucas and Takeo Kanade, whose 1981 paper "An Iterative Image Registration Technique with an Application to Stereo Vision" introduced the method now also used for optical flow. Correlation suits images from the same sensor, and mutual information, introduced for medical registration in the mid-1990s, handles different modalities such as MRI against CT, where the same tissue appears with different brightness. An optimizer, often gradient descent across a coarse-to-fine image pyramid, searches for the best parameters, and toolkits such as the SimpleITK registration framework expose the metric, optimizer, transformation, and interpolator as separate components.

Deep learning methods such as VoxelMorph, published by Guha Balakrishnan and colleagues in 2019, train a convolutional neural network to predict a deformation field in a single forward pass, replacing a per-pair optimization that takes minutes or longer with a prediction that takes under a second on a GPU.

Choosing a Transformation Model#

The transformation should match what actually differs between the images:

• Translation and rigid: Shifts, plus rotation in the rigid case, with distances preserved. Common for scans of the same rigid object.
• Similarity and affine: Similarity adds uniform scaling, and affine adds non-uniform scaling and shear, one transformation for the whole image. Suits satellite images taken at slightly different scales.
• Projective (homography): Models perspective changes for planar scenes or rotating cameras, as in document scanning and panoramas.
• Deformable: Lets each region move independently, as needed for soft tissue that changes shape between scans.

More flexible models can fit almost anything, including real changes the analysis should detect, so registration usually starts with the simplest plausible model and adds flexibility only where alignment error remains. Scenes with depth and a moving camera show parallax that no single 2D transformation removes, and their alignment relies on epipolar geometry and 3D reconstruction instead.

Where Registration Is Used#

In remote sensing and satellite image analysis, registration is the first step of change detection: images of the same area taken months apart must line up before a model can flag new construction or cleared forest, or a slight shift will look like change along every road edge. In medical image analysis, aligning scans from separate visits lets clinicians measure whether a tumor has grown, aligning PET with CT combines metabolic activity with anatomy, and registering a labeled atlas to a new scan transfers its anatomical labels, a classic route to image segmentation of brain structures.

Video applications register consecutive frames to estimate camera motion. Stabilization removes that motion, and trackers use it so that footage from a moving camera does not confuse camera motion with object motion. The default TrackTrack tracker in Ultralytics YOLO object tracking registers each frame to the previous one by following corner features with sparse optical flow at half resolution and fitting a similarity transform with RANSAC, which keeps tracks from a drone or vehicle camera attached to the right objects.