Deformable Attention

Deformable attention is an attention mechanism for computer vision in which each query attends to a small, fixed number of sampling points around a reference location instead of to every position of a feature map. The query itself predicts where those points sit, as offsets from the reference, and how much each one counts, so the cost grows with the number of queries and points rather than with image resolution. Xizhou Zhu, Jifeng Dai, and colleagues at SenseTime, the University of Science and Technology of China, and the Chinese University of Hong Kong introduced it in Deformable DETR, an oral paper at ICLR 2021. It fixed the two weaknesses of the original DETR detection transformer: full attention limited it to low-resolution features, and training took 500 epochs. Deformable DETR needed 10 times fewer epochs and detected small objects better.

How Deformable Attention Works#

The idea extends deformable convolution, also co-developed by Dai, which learns offsets for the taps of a convolution kernel. For each query with a reference point, every attention head uses two linear layers on the query features: one predicts K sampling offsets and the other K attention weights, normalized with a softmax. The value features at each shifted point are read with bilinear interpolation, since offsets are fractional, and the head returns their weighted sum. The paper uses 8 heads and K = 4 points, so a query reads 32 locations per feature map where standard self-attention would score every pixel. A single-head, single-scale version fits in a few lines of PyTorch:

import torch
import torch.nn.functional as F

torch.manual_seed(0)
dim, n_points, n_queries = 32, 4, 100
value = torch.randn(1, dim, 64, 64)  # a 64x64 feature map: 4,096 positions
query = torch.randn(n_queries, dim)
ref = torch.rand(n_queries, 2) * 2 - 1  # reference points in grid_sample's [-1, 1] coordinates

# Each query predicts where to look (offsets) and how much each sample counts (weights)
to_offsets, to_weights = torch.nn.Linear(dim, n_points * 2), torch.nn.Linear(dim, n_points)
offsets = 0.1 * to_offsets(query).view(n_queries, n_points, 2)
weights = to_weights(query).softmax(dim=-1)

# Bilinearly sample the value map at reference + offset, then take the weighted sum
grid = (ref[:, None] + offsets)[None]  # (1, n_queries, n_points, 2)
sampled = F.grid_sample(value, grid, align_corners=False)[0]  # (dim, n_queries, n_points)
out = (sampled * weights).sum(dim=-1).T  # (n_queries, dim)
print(f"Output {tuple(out.shape)} from {n_queries * n_points} samples, not {n_queries * 64 * 64:,} dense scores")

It prints Output (100, 32) from 400 samples, not 409,600 dense scores: each of the 100 queries reads 4 of the 4,096 positions. Training adjusts the offset and weight layers, so the points move toward the parts of the image that help the task.

Multi-Scale Deformable Attention#

Objects appear at very different sizes, so Deformable DETR samples points on several feature maps at once. With L levels, each head takes K points on every level and normalizes the weights over all L × K points together, which mixes fine and coarse features in one step and replaced the feature pyramid network (FPN) in its encoder. The encoder applies it as self-attention, with every pixel of every level as a query, and the decoder applies it as cross-attention from object queries to the image features.

RT-DETR, Baidu's real-time detection transformer available in the Ultralytics package, uses multi-scale deformable attention as the cross-attention of its six decoder layers: each of its 300 object queries samples 4 points per head in 8 heads on the stride-8, stride-16, and stride-32 feature maps, 96 locations in all, where full cross-attention over those maps of a 640-pixel image would score 8,400 positions.

Applications#

• Object detection: Deformable DETR's successors, including DINO and RT-DETR, keep deformable cross-attention in their decoders, which lets them read high-resolution features for small objects.
• Autonomous vehicles: BEVFormer lifts images from several car cameras into a bird's-eye-view grid, where each grid cell samples deformable points in the camera views that see it.
• Panoptic segmentation: Mask2Former builds its multi-scale pixel features with deformable attention before predicting masks.

Deformable Attention vs. Related Terms#

• Sliding window attention: Attends to a fixed neighborhood set by position, the same for every input.
• Sparse attention: Scores a fixed subset of token pairs chosen in advance.
• Flash attention: Computes ordinary dense attention faster by tiling it in on-chip GPU memory, with the same result as standard attention.
• Vision transformer (ViT) attention: Compares every image patch with every other patch, so its cost grows with the square of the number of patches.