Sliding Window Attention

Sliding window attention (SWA) limits each token to attending over a fixed number of neighboring tokens, in a causal language model the w most recent ones, instead of over the whole sequence as in full self-attention. Attention compute then grows linearly with sequence length instead of quadratically, and a generating model only needs to keep the keys and values of the last w tokens in its KV cache. Longformer made the sliding window the core of its attention in 2020, and Mistral AI's Mistral 7B brought it to a widely used open large language model (LLM) in 2023, with a window of 4,096 tokens.

How the Window Grows with Depth#

One layer of SWA sees only w tokens, but transformer layers stack. At layer 2, a token reads hidden states that each already summarize w tokens from layer 1, so information can travel about w − 1 more positions with every layer, just as stacked convolutions widen the receptive field of a convolutional neural network (CNN). In PyTorch, the window is a boolean attention mask, and multiplying that mask by itself traces how far information travels:

import torch

seq_len, window = 16, 4
pos = torch.arange(seq_len)
offset = pos[:, None] - pos[None, :]

# Causal sliding window: each token attends to itself and the window - 1 tokens before it
mask = (offset >= 0) & (offset < window)
print(f"Scores per layer: {mask.sum().item()} with the window, {(offset >= 0).sum().item()} with full causal attention")

# Stacking layers widens the reach: each layer adds window - 1 more tokens of history
reach = mask
for layer in range(1, 5):
    print(f"Layer {layer}: last token draws on {reach[-1].sum().item()} tokens")
    reach = (mask.int() @ reach.int()) > 0

The window computes 58 scores per layer for 16 tokens instead of 136, and the last token's reach grows by 3 tokens per layer, from 4 to 7, 10, and 13. The same mask can be passed as attn_mask to PyTorch's scaled_dot_product_attention. For Mistral 7B, 32 layers with a 4,096-token window give a theoretical attention span of about 131K tokens, although information from far back has to survive many hops to arrive.

Mistral 7B also stores its keys and values in a rolling buffer of w slots, writing position i into slot i mod w and overwriting the oldest entry. On a 32K-token sequence, this cut cache memory eight times without hurting quality, and the fixed cache size keeps memory flat however long generation runs.

Interleaving Local and Global Layers#

Pure windows cannot copy a fact from far back in one step, so recent models mix SWA layers with full global layers. Google's Gemma 2 alternated them one to one with a 4,096-token window, and Gemma 3 uses five local layers for every global layer with a window of only 1,024 tokens, which keeps the KV cache small across a context window of up to 128K tokens. Only the global layers store keys and values for the whole context, so five of every six layers hold just 1,024 tokens each.

Vision transformers use the same mix for high-resolution images, where every patch is a token. Meta's SAM 3, which runs in the Ultralytics package for image segmentation, attends within non-overlapping 24 × 24 windows of patches in 28 of the 32 blocks of its image encoder and globally across the full 72 × 72 patch grid of a 1,008-pixel image in blocks 7, 15, 23, and 31.

SWA vs. Related Terms#

• Swin Transformer: Splits the image into fixed, non-overlapping windows that every token in the window shares, and shifts the grid between layers so neighboring windows exchange information.
• Sparse attention: The broader family of fixed patterns. SWA is its simplest member, and others add dilated windows, global tokens, or random links.
• Attention sinks: The first tokens of a sequence, which absorb spare attention weight. When a window is applied at inference to a model trained with full attention, evicting them breaks generation, and StreamingLLM keeps them in the cache next to the window.
• Deformable attention: Each query attends to a few sampling points whose offsets it learns, so the neighborhood depends on the content.
• Flash attention: A GPU kernel for exact attention. Its implementations accept a window size, so SWA models skip the blocks outside the window entirely.