Self-Attention

Self-attention is the form of attention in which the queries, keys, and values all come from the same sequence, so every token builds its new representation from every other token in that sequence. One self-attention layer can connect the first and last words of a paragraph, or opposite corners of an image, in a single step, where a recurrent neural network (RNN) needs one step per position and a convolutional neural network (CNN) needs many stacked layers to grow its view. Stacked with feed-forward layers, it forms the transformer introduced in the 2017 paper Attention Is All You Need, which underlies large language models (LLMs), Vision Transformers (ViT), and transformer-based detectors.

How Self-Attention Works#

Each input token, represented as an embedding, is multiplied by three learned matrices to give a query, a key, and a value. The output for token i is a weighted average of every token's value vector, with weights given by a softmax over the scaled dot products of token i's query with all the keys, the scaled dot-product attention described on the attention mechanism page. Because the queries, keys, and values are projections of the same tokens, a sequence of n tokens produces an n × n weight matrix. In "The animal didn't cross the street because it was too tired," the row for "it" can put most of its weight on "animal" and pull that word's meaning into its own representation.

Multi-head self-attention runs several of these computations in parallel on smaller projections and concatenates the results, so different heads can track different relations, such as syntax in one head and coreference in another. The original transformer used 8 heads of 64 dimensions each in a 512-dimensional model.

Order Blindness and Causal Masking#

Self-attention treats its input as a set. Shuffling the tokens shuffles the outputs the same way and changes nothing else, so models add position information, such as sinusoidal embeddings in the original transformer or rotary position embedding (RoPE) in most current LLMs. Encoders such as BERT let every token attend in both directions. Decoder-only LLMs apply a causal mask so each token attends only to itself and earlier tokens, which lets a model generate text left to right and reuse the keys and values of earlier tokens from a KV cache. PyTorch's MultiheadAttention shows both properties when the same tensor is passed as query, key, and value:

import torch

torch.manual_seed(0)
tokens = torch.randn(1, 5, 16)  # one sequence of 5 tokens with 16 features each
attn = torch.nn.MultiheadAttention(embed_dim=16, num_heads=2, batch_first=True)

# Self-attention: queries, keys, and values are all projections of the same tokens
out, weights = attn(tokens, tokens, tokens)
print("Attention weights shape:", tuple(weights.shape))

# Shuffling the tokens only shuffles the outputs, so word order is invisible without positions
perm = torch.tensor([3, 0, 4, 1, 2])
out_shuffled, _ = attn(tokens[:, perm], tokens[:, perm], tokens[:, perm])
print("Same outputs after shuffling:", torch.allclose(out_shuffled, out[:, perm], atol=1e-6))

# A causal mask blocks attention to later tokens, as in a decoder-only LLM
mask = torch.triu(torch.ones(5, 5, dtype=torch.bool), diagonal=1)
_, causal_weights = attn(tokens, tokens, tokens, attn_mask=mask)
print(causal_weights[0].detach().round(decimals=2))

It prints a (1, 5, 5) weight shape, True for the shuffle test, and a lower-triangular matrix: the first token puts weight 1.00 on itself, the second splits 0.31 and 0.69 between the first two tokens, and every entry above the diagonal is 0.00.

Self-Attention vs. Related Terms#

• Attention mechanism: The general operation that weights values by query-key similarity. Self-attention is the case where all three come from one sequence.
• Cross-attention: Queries come from one sequence and keys and values from another. A translation decoder attends to the encoded source sentence, the object queries of a DETR decoder attend to image features, and a text-to-image diffusion model attends to the prompt.
• Convolution: Applies the same small filter at every position, so each output sees a fixed local window with weights that do not depend on the content. Self-attention computes its weights from the content and covers the whole input.

The original encoder-decoder transformer combines bidirectional self-attention, causal self-attention, and cross-attention:

Cost and Efficient Variants#

Scoring every pair of tokens makes time and memory grow with the square of the sequence length: 1,000 tokens give one million scores per head and 100,000 tokens give ten billion. FlashAttention computes exact self-attention in tiles without storing the full matrix, grouped-query attention (GQA) shares keys and values across heads to shrink the KV cache, and sliding-window, sparse, and linear attention reduce the number of pairs. In vision, the token count follows image resolution, so many detectors apply self-attention only where feature maps are smallest. YOLO26 runs multi-head self-attention only in the C2PSA block at the end of its backbone and one attention-enabled C3k2 block in the neck, both on the 20×20 stride-32 feature map of a 640-pixel image, or 400 tokens. Its attention module computes queries, keys, and values with one 1×1 convolution over that map and adds position information with a 3×3 depthwise convolution applied to the values.