Linear Attention

Linear attention is a family of attention mechanisms whose cost grows linearly with sequence length, in place of the quadratic cost of standard softmax self-attention. Softmax attention scores every query against every key, so N tokens produce an N × N score matrix per head, and a transformer that generates text keeps a KV cache that grows with every token. Linear attention replaces the softmax similarity with a dot product of feature maps, φ(q)·φ(k), which lets the model regroup the matrix products and summarize all past keys and values in a fixed-size state. A 2020 paper by Katharopoulos et al., Transformers are RNNs, gave the method its common form, and gated variants now run in production large language models (LLMs) alongside standard attention layers.

How Linear Attention Works#

Softmax attention computes softmax(QKᵀ)V, and because exp(q·k) does not factor into a query part and a key part, every query-key pair has to be scored. With a feature map φ, such as elu(x) + 1, the output becomes φ(Q)(φ(K)ᵀV), normalized by φ(Q) times the sum of the φ(K) rows. Matrix multiplication is associative, so φ(K)ᵀV can be computed first, giving a d × d matrix, where d is the head dimension, however many tokens there are. Time drops from O(N²d) to O(Nd²), which pays off when sequences are much longer than the head dimension.

For causal models, the same algebra turns attention into a recurrent neural network (RNN) with a matrix-valued state. Each new token adds the outer product φ(k)vᵀ to the state, and its query reads from that state, so text generation takes constant memory and constant time per token. Training uses the parallel form, or chunkwise kernels such as those in the flash-linear-attention library, so a GPU still processes whole sequences at once. In PyTorch, the two forms give the same outputs:

import torch
import torch.nn.functional as F

torch.manual_seed(0)
n, d = 2048, 64
q, k, v = torch.randn(3, n, d).unbind(0)
phi_q, phi_k = F.elu(q) + 1, F.elu(k) + 1  # positive feature map in place of softmax

# Parallel form: an n x n causal score matrix, as in softmax attention
scores = torch.tril(phi_q @ phi_k.T)
parallel = (scores @ v) / scores.sum(dim=-1, keepdim=True)

# Recurrent form: a fixed d x d state that each new token updates
state, norm, outputs = torch.zeros(d, d), torch.zeros(d), []
for t in range(n):
    state += torch.outer(phi_k[t], v[t])
    norm += phi_k[t]
    outputs.append(phi_q[t] @ state / (phi_q[t] @ norm))
recurrent = torch.stack(outputs)

print("Same outputs:", torch.allclose(parallel, recurrent, atol=1e-4))
print(f"Recurrent state: {state.numel():,} numbers; KV cache after {n:,} tokens: {2 * n * d:,} numbers")

It prints Same outputs: True. The 64 × 64 state holds 4,096 numbers however long the sequence grows, while one head's KV cache after 2,048 tokens holds 262,144 numbers and grows by 128 with every new token.

DeltaNet, Gating, and Hybrid Models#

A fixed-size state is lossy memory. Plain linear attention only adds to it, so old associations pile up and interfere, and recall of specific facts from a long context window falls behind softmax attention. Two fixes now dominate. Gating multiplies the state by a learned decay before each update so the model can forget, a mechanism shared with state space models (SSMs) such as Mamba-2, whose authors showed that the two families are dual forms of each other. The delta rule, from work linking linear transformers to fast weight programmers, reads the value currently stored for a key and writes only the difference between the new value and that stored prediction. DeltaNet made delta-rule training parallel across the sequence in 2024, Gated DeltaNet (ICLR 2025) added a decay gate, and Moonshot AI's Kimi Linear refines that gate to act on each channel separately.

Production models interleave these layers with full attention so a few softmax layers keep exact recall. Qwen3-Next and the Qwen3.5 family repeat a block of three Gated DeltaNet layers followed by one gated softmax attention layer, and Kimi Linear uses a similar 3:1 mix of Kimi Delta Attention and multi-head latent attention:

Linear Attention vs. Related Methods#

• Self-attention: The exact softmax form that scores all pairs of tokens, with quadratic time and a KV cache that grows with the sequence.
• Flash attention: A GPU kernel that computes exact softmax attention in tiles held in on-chip memory. It cuts memory traffic, and the number of operations stays quadratic.
• Sparse attention: Keeps the softmax and scores only selected pairs, such as a local band of recent tokens.
• Grouped-query attention (GQA): Shares keys and values across query heads to shrink the KV cache by a constant factor while keeping full softmax attention.

Where Linear Attention Is Used#

Long-context language models are the largest use, because the linear layers in a hybrid keep a constant-size state during decoding and only the few softmax layers carry a growing KV cache. Vision is the second, since the number of tokens grows with image resolution. EfficientViT uses ReLU-based linear attention for high-resolution semantic segmentation, and NVIDIA's SANA builds a linear diffusion transformer that generates images up to 4096 × 4096 pixels. In object detection, attention usually runs only where feature maps are smallest. YOLO26 keeps standard softmax attention and runs it only in its C2PSA block and one attention-enabled C3k2 block, both on the 20 × 20 stride-32 feature map of a 640-pixel image, so each attention layer sees 400 tokens and a full score matrix of 160,000 entries per head.