Scale-Invariant Feature Transform (SIFT)

Scale-Invariant Feature Transform (SIFT) is a classic computer vision algorithm that finds distinctive local keypoints in an image and describes each one with a 128-number vector that stays largely the same when the image is scaled or rotated. Matching these descriptors lets software link two photos of the same scene taken from different distances and angles, which is how overlapping photos are joined into a panorama. SIFT is a hand-designed form of feature extraction, unlike the learned feature maps of a convolutional neural network.

Origin and Core Idea#

David Lowe at the University of British Columbia introduced SIFT in the 1999 ICCV paper Object Recognition from Local Scale-Invariant Features and gave the full treatment in "Distinctive Image Features from Scale-Invariant Keypoints" in the International Journal of Computer Vision in 2004. SIFT features are invariant to scale and rotation, and they match robustly across a substantial range of affine distortion, 3D viewpoint change, noise, and illumination change. The method was patented, and the patent expired in 2020. OpenCV moved SIFT from its contrib repository into the main library in version 4.4.0.

An object has no fixed size in a photo, so a detector must look for structure at every size. SIFT builds a scale space by repeatedly applying Gaussian blur at increasing strengths and downsampling in groups called octaves.

How SIFT Works#

The algorithm runs in four stages:

1. Scale-space extrema detection: Differences between neighboring blur levels (difference of Gaussians) approximate a blob detector. A pixel is a candidate if it is the maximum or minimum among its 26 neighbors across space and scale.
2. Keypoint localization: Candidates are refined to sub-pixel accuracy. Low-contrast points and points lying along edges are discarded because they are unstable.
3. Orientation assignment: A histogram of local gradient directions around each keypoint sets a dominant orientation, and everything after is measured relative to it. This gives rotation invariance.
4. Descriptor construction: The neighborhood is split into a 4 x 4 grid, and each cell holds an 8-bin gradient histogram. That gives 4 x 4 x 8 = 128 values, normalized to reduce sensitivity to lighting.

Matching compares descriptors by Euclidean distance. Lowe's ratio test keeps a match only when the best neighbor is clearly closer than the second best. A k-nearest neighbors search with k = 2 supplies both distances.

import cv2
from ultralytics.utils import ASSETS

# Load a sample image in grayscale and a half-size copy
img = cv2.imread(str(ASSETS / "bus.jpg"), cv2.IMREAD_GRAYSCALE)
small = cv2.resize(img, None, fx=0.5, fy=0.5)

# Detect keypoints and compute 128-dimensional descriptors
sift = cv2.SIFT_create()
kp1, des1 = sift.detectAndCompute(img, None)
kp2, des2 = sift.detectAndCompute(small, None)

# Match half-size descriptors to full-size ones, then apply Lowe's ratio test
pairs = cv2.BFMatcher().knnMatch(des2, des1, k=2)
good = [m for m, n in pairs if m.distance < 0.8 * n.distance]
print(f"{len(good)} of {len(kp2)} half-size keypoints matched at full size")

The printed count shows how many keypoints found at half size are recognized again in the full-size image, which is scale invariance at work.

Applications#

SIFT became the default local feature for geometric vision tasks:

• Panorama stitching: Brown and Lowe's automatic panoramic stitching (IJCV 2007) matches SIFT features between overlapping photos and estimates a homography to align them, an alignment step called image registration.
• 3D reconstruction: Photo Tourism (Snavely, Seitz, and Szeliski, 2006) matches SIFT features across tourist photos and uses epipolar geometry to recover camera poses and sparse 3D structure.
• Robot localization: Se, Lowe, and Little (2001) used SIFT landmarks for mobile robot localization and mapping, an ancestor of modern visual SLAM.
• Instance recognition: Lowe's 2004 paper recognizes a toy train and a frog in a cluttered, partly occluded scene from as few as 3 matched features.

In Ultralytics YOLO object tracking, BoT-SORT compensates for camera motion between frames, and setting gmc_method: sift in the tracker config uses SIFT matches for that estimate instead of the default sparse optical flow.

SIFT Compared with Related Methods#

SIFT matches the same physical points across views, which suits correspondence and geometry. Object detection instead predicts the category and location of objects. Template matching slides a fixed patch and breaks when scale or rotation changes.

SIFT is slow compared with ORB, a binary descriptor introduced by Rublee and colleagues in 2011 as an alternative to SIFT and SURF. It also struggles on textureless surfaces and under very large viewpoint changes, and learned models have largely replaced it for category-level recognition.

In Lowe's 2004 tests, rejecting matches with a distance ratio above 0.8 eliminated 90% of false matches while discarding fewer than 5% of correct ones, on images with random scale and rotation changes, 30 degrees of depth rotation and 2% noise, matched against 40,000 keypoints.