Longformer

Longformer is a transformer for long documents that replaces full self-attention with sliding-window attention around every token plus global attention on a few task-chosen tokens, so its cost grows linearly with input length instead of quadratically. Iz Beltagy, Matthew Peters, and Arman Cohan of the Allen Institute for AI introduced it in the 2020 paper Longformer: The Long-Document Transformer. Encoders such as BERT and RoBERTa read at most 512 tokens, so longer texts had to be cut into chunks that lose the links between them. The released Longformer models read 4,096 tokens in one pass, a context window eight times longer.

How Longformer Attention Works#

Each layer combines local and global patterns inside one attention mechanism:

• Sliding window attention: Every token attends to a fixed window of neighbors, 256 on each side in the released models.
• Dilated sliding window: Gaps between the attended positions let a head see further at the same cost. The authors used dilation on a few heads in the upper layers of their character-level language models.
• Global attention: Selected tokens attend to every position and every position attends to them, through separate query, key, and value projections. The task picks them: the [CLS] token for classification and the question tokens for question answering.

Stacking layers widens the reach of the window the way stacked convolutions grow the receptive field of a convolutional neural network (CNN): l layers with window w cover l × w tokens. Each layer computes about n × w attention scores instead of n × n. Counting the pairs in a Longformer mask makes the difference visible:

import torch

window = 512  # Longformer-base: 256 tokens on each side of every token

for n in (1024, 2048, 4096):
    pos = torch.arange(n)
    # Sliding window: each token attends to neighbors within window / 2 positions
    mask = (pos[:, None] - pos[None, :]).abs() <= window // 2
    # Global attention on token 0 ([CLS]): it sees every token and every token sees it
    mask[0, :] = True
    mask[:, 0] = True
    print(f"{n} tokens: full attention {n * n / 1e6:.1f}M scores, Longformer {mask.sum().item() / 1e6:.1f}M")

It prints 0.5M, 1.0M, and 2.0M Longformer scores against 1.0M, 4.2M, and 16.8M for full attention: each doubling of the input doubles Longformer's attention work and quadruples the work of full attention. Real implementations compute only the band of scores, which is where the memory savings come from.

Training and Results#

Longformer starts from the RoBERTa checkpoint. The authors copied its 512 learned position embeddings eight times to cover 4,096 positions and continued masked language model pretraining on long documents. After fine-tuning, it consistently outperformed RoBERTa on long-document tasks and set state-of-the-art results at the time on the WikiHop and TriviaQA question answering benchmarks, while the character-level language model set new results on text8 and enwik8.

The Longformer-Encoder-Decoder (LED) adds a decoder for text summarization and other text generation tasks, demonstrated on summarizing full scientific papers from arXiv. Hugging Face Transformers implements both models and takes the global tokens as a global_attention_mask. Domain versions followed, such as Clinical-Longformer, which was pretrained on large clinical text corpora and evaluated on 10 tasks including named entity recognition, question answering, and document classification.

Longformer vs. Related Models#

• Transformer-XL: Reads a long text segment by segment and lets each segment attend to cached hidden states of the previous one, extending context through recurrence for left-to-right generation.
• BigBird: A Google model from the same year that adds random attention links to window and global attention and also reads 4,096 tokens.
• Sparse attention: The broader family of fixed attention patterns that Longformer belongs to.
• Flash attention: Computes exact full attention in on-chip memory tiles, so memory grows linearly with length while the number of scores stays quadratic.

Local and Global Attention in Vision#

High-resolution images pose the same problem, because a vision transformer (ViT) turns an image into thousands of patch tokens. The Swin Transformer answers it with shifted local windows, and Meta's Segment Anything Model (SAM), which runs in the Ultralytics package, mixes windowed and global attention block by block for image segmentation: its ViT-B image encoder attends within non-overlapping 14 × 14 windows of its 64 × 64 patch grid in 8 of its 12 blocks and globally across all 4,096 patches in blocks 2, 5, 8, and 11.