Attention Sinks

An attention sink is a token that receives a large share of attention weight in many heads and layers although its content is irrelevant to the query, most often the first token of the sequence. Guangxuan Xiao and colleagues at MIT, Meta AI, and Carnegie Mellon University named the effect in Efficient Streaming Language Models with Attention Sinks, published at ICLR 2024. They showed that large language models (LLMs) such as Llama 2, MPT, Falcon, and Pythia break down when a sliding KV cache evicts the first tokens, and that keeping four of them restores stable generation.

Why Sinks Form#

Self-attention turns each query's scores into weights with a softmax, and those weights always sum to one. When no token in the context is relevant to a head, the weight still has to go somewhere. In a causal language model, the first tokens are visible to every later token, so training turns them into the default destination for weight a head does not need. In the StreamingLLM analysis of Llama 2, every layer beyond the bottom two concentrated attention on the initial tokens across all heads. A sink logit added to the softmax shows the effect:

import torch

torch.manual_seed(0)
values = torch.randn(6, 4)
# A query that matches none of the 6 tokens: every score is close to zero
scores = torch.tensor([0.1, -0.2, 0.0, 0.2, -0.1, 0.0])

# Softmax must hand out a total weight of 1, so the output mixes all six values
weights = scores.softmax(dim=-1)
print(f"No sink: weight on tokens {weights.sum():.2f}, output norm {(weights @ values).norm():.2f}")

# A sink logit joins the softmax but carries no value, so it absorbs the spare weight
sink = torch.tensor([3.0])
weights = torch.cat([sink, scores]).softmax(dim=-1)[1:]
print(f"With sink: weight on tokens {weights.sum():.2f}, output norm {(weights @ values).norm():.2f}")

Without a sink, the head spreads its full weight over six irrelevant values and returns an output of norm 0.83. With a sink logit of 3, the sink takes 77% of the weight and the output shrinks to 0.19, so the head does almost nothing for this query. OpenAI's open-weight gpt-oss models build this in with a learned sink logit for every attention head, so each head has a place for spare weight that adds nothing to its output.

Streaming Generation with StreamingLLM#

A model that generates indefinitely outgrows any context window and cannot keep every token in its KV cache, and plain window attention, which keeps only the most recent tokens, fails as soon as the first tokens drop out. StreamingLLM keeps the keys and values of four initial tokens permanently next to a rolling window of recent ones, and assigns positions by place in the cache rather than in the original text, re-applying rotary position embeddings (RoPE) at every step:

This kept language modeling stable over 4 million tokens and more, and ran up to 22.2 times faster than a sliding window that recomputes the cache for every new token. The window still bounds what the model can recall: tokens evicted from the middle are gone for good, so a streaming chatbot or AI agent remembers only its most recent turns. The authors also pretrained models with one dedicated, learnable placeholder token at the start of every sample, which took over the sink role on its own.

Attention Sinks vs. Related Terms#

• Sliding window attention: Limits each token to recent neighbors. Applied at inference to a model trained with full attention, it fails once the sinks leave the window.
• Context rot: The decline in accuracy as a prompt grows while it still fits the context window.

Sinks in Vision Models#

Vision transformers (ViTs) show a related behavior. In Vision Transformers Need Registers, Meta and Inria researchers found tokens with unusually large norms on low-information background patches of supervised and self-supervised ViTs, repurposed for internal computation. Adding a few extra register tokens to the input absorbed that role, removed the artifacts from feature maps and attention maps, and improved dense prediction.

Meta's SAM 2, which runs in the Ultralytics package, budgets its video memory with a similar keep-the-anchor-plus-a-window plan: for each new frame, its memory attention reads the memory bank entries of the user-prompted frames, kept for the whole video, plus those of the 6 most recent frames, so the attention cost per frame stays fixed however long the video runs.