Swin Transformer

Swin Transformer, short for Shifted Window Transformer, is a hierarchical vision transformer that computes self-attention inside small local windows and shifts those windows between consecutive layers. Ze Liu and colleagues at Microsoft Research Asia introduced it in the 2021 paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, which won the Marr Prize for best paper at ICCV 2021. Local windows make the cost of attention grow linearly with the number of pixels instead of quadratically, and merging patches stage by stage produces multi-scale feature maps like those of a convolutional neural network. Together these made transformers practical general-purpose backbones for object detection and segmentation on high-resolution images, beyond classification.

How Shifted Windows Work#

Swin first splits the image into 4×4-pixel patches and projects each one to a token embedding, so a 224×224 image becomes a 56×56 grid of tokens with 96 channels in the Tiny model. Global attention over all 3,136 tokens would compare every token with every other one; Swin partitions the grid into non-overlapping windows of 7×7 tokens and computes attention only within each window. The paper gives the cost of global attention as 4hwC² + 2(hw)²C for an h×w grid with C channels, against 4hwC² + 2M²hwC for windows of M×M tokens, which is linear in hw.

Windows alone would never exchange information, so every second block shifts the window grid by half a window, 3 tokens, and the new windows straddle the old boundaries. To keep the number of windows constant, the implementation rolls the feature map cyclically and masks attention between tokens that were not neighbors in the original image. Each block also adds a learned relative position bias to the attention scores.

Between stages, a patch-merging layer concatenates each 2×2 group of neighboring tokens and projects them linearly, halving the resolution and doubling the channels. The four stages output feature maps at 1/4, 1/8, 1/16, and 1/32 of the input resolution, the same strides as ResNet, so Swin drops into existing detection and segmentation heads such as a feature pyramid network. The torchvision implementation shows both mechanisms:

import torch
from torchvision.models import swin_t

# Swin-T architecture; random weights are enough to inspect its structure
model = swin_t().eval()

# Each pair of blocks uses 7x7 windows, the second shifted by 3 patches
regular, shifted = model.features[1]
print(f"Window {regular.attn.window_size}, shift {regular.attn.shift_size} then {shifted.attn.shift_size}")

# features alternates stages with patch merging; stage outputs are (batch, height, width, channels)
x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    for i, layer in enumerate(model.features):
        x = layer(x)
        if i % 2 == 1:
            print(f"Stage {i // 2 + 1}: {tuple(x.shape)}")

It prints Window [7, 7], shift [0, 0] then [3, 3], followed by stage outputs of (1, 56, 56, 96), (1, 28, 28, 192), (1, 14, 14, 384), and (1, 7, 7, 768): each stage halves the grid and doubles the channels, giving the pyramid that dense prediction heads expect.

Swin Transformer vs. ViT and CNNs#

The original ViT cuts images into 16×16 patches, keeps a single resolution through all layers, and attends globally, which suits image classification but becomes expensive for the high-resolution inputs of detection and segmentation. Swin reintroduces two priors of convolutional networks, locality and a feature hierarchy, while keeping the input-dependent weights of the attention mechanism. Its local windows differ from sliding window attention, where every token attends to its own neighborhood, because Swin's windows are fixed partitions that tokens share, which maps efficiently onto GPU matrix multiplications.

At release, the largest model, Swin-L pretrained on ImageNet-22K, reached 87.3% top-1 accuracy on ImageNet-1K, 58.7 box AP and 51.1 mask AP on COCO test-dev, and 53.5 mIoU on ADE20K, the best published results on those detection and semantic segmentation benchmarks. Its success also pushed convolutional research forward: Meta's ConvNeXt (2022) modernized ResNet with design choices taken from Swin and matched or exceeded its accuracy.

Swin can serve as the backbone of a custom Ultralytics model: the TorchVision module in a model YAML file loads torchvision's swin_t with its pretrained ImageNet weights, and a YOLO detection head trains on the single stride-32 feature map it returns.

Variants and Applications#

• Swin Transformer V2: Scaled the design to 3 billion parameters and images up to 1,536×1,536 pixels in 2022, using scaled cosine attention and a log-spaced continuous position bias to keep training stable.
• Video action recognition: Video Swin Transformer extends the windows to three dimensions, across space and time, to classify actions in video clips.
• Super-resolution and restoration: SwinIR uses Swin blocks for image upscaling, denoising, and JPEG artifact removal.
• Medical image analysis: Swin-Unet and Swin UNETR combine Swin encoders with U-Net-style decoders, and Swin UNETR segments brain tumors in 3D MRI volumes.