Differential Transformer

The Differential Transformer, or DIFF Transformer, is a transformer architecture whose attention heads compute two softmax attention maps and subtract one from the other, canceling the weight that both maps spread over irrelevant context. Tianzhu Ye, Li Dong, Furu Wei, and colleagues at Microsoft Research and Tsinghua University introduced it in October 2024 in Differential Transformer, an oral paper at ICLR 2025. The design borrows from differential amplifiers and noise-canceling headphones, which subtract two signals so that the noise they share drops out. In language modeling, a DIFF Transformer needed about 65% of the parameters or training tokens of a standard transformer to reach the same loss: a 6.8-billion-parameter model matched an 11-billion-parameter transformer.

How Differential Attention Works#

Standard self-attention gives every token a nonnegative weight, so some weight always lands on tokens that do not matter, which the authors call attention noise. Differential attention splits each head's query and key projections into two halves, computes a softmax map from each pair, and combines the values as (map1 − λ × map2) × V. Both maps assign weight to the same irrelevant tokens, so the subtraction cancels it while weight that only map1 assigns survives. Because each map sums to 1, every row of the result sums to 1 − λ, and individual weights can be negative:

import torch

torch.manual_seed(0)
n, d, lam = 6, 8, 0.8  # 6 tokens, head size 8, lambda near the paper's 0.8 initialization
q1, k1, q2, k2 = torch.randn(4, n, d)
v = torch.randn(n, 2 * d)  # values are twice as wide as each query-key pair

# Two ordinary softmax attention maps from separate query and key projections
map1 = torch.softmax(q1 @ k1.T / d**0.5, dim=-1)
map2 = torch.softmax(q2 @ k2.T / d**0.5, dim=-1)

# Subtracting cancels weight that both maps place on the same tokens
diff = map1 - lam * map2
out = diff @ v
print(f"Each row sums to {diff.sum(dim=-1)[0]:.2f}; {(diff < 0).sum().item()} of {n * n} weights are negative")

It prints Each row sums to 0.20; 13 of 36 weights are negative, two properties no softmax attention map can have. In the real model, λ is learned through four small vectors and starts at 0.8 − 0.6 × exp(−0.3 × (l − 1)) for layer l, so deeper layers start closer to 0.8. Each head's output is normalized on its own, which the paper writes as GroupNorm, then scaled by (1 − λ at initialization) to keep gradients the same size as in a standard transformer. The model uses half as many heads, each with values twice as wide, so its parameter count matches the baseline. Each map is ordinary softmax attention, so implementations reuse flash attention kernels, and training throughput on H100 GPUs was 6% to 12% lower than for a transformer of the same size.

Reported Benefits#

In a needle-in-a-haystack test with 6 needles and 2 questions in a 4K-token context window, the DIFF Transformer scored 0.85 against 0.55 for a standard transformer, and the paper's attention analysis traced this to focus: it put 0.27 to 0.40 of its attention on the answer, against 0.03 to 0.09 for the transformer. It also produced fewer hallucinations in text summarization and question answering, and in few-shot in-context learning it was more accurate and less sensitive to the order of the examples.

Activation outliers shrank as well. Its largest attention logit was 38.8, against 318.0 for a standard transformer, and its largest hidden-state activation was 1,688 against 3,609. With attention logits quantized to 4 bits, it matched the accuracy of a standard transformer at 6 bits and beat a 4-bit transformer by about 25% in accuracy.

That result matters for model quantization, because a few extreme activations set the range that every other value must share. The same applies to vision models: when Ultralytics YOLO26 is exported with quantize=8, the exporter calibrates activation ranges on images from the data dataset and recommends at least 300 of them, while quantize="w8a16" keeps weights at 8 bits and activations at 16 for LiteRT, Core ML, and QNN exports.

DIFF Transformer vs. Related Terms#

• Diffusion Transformer (DiT): A transformer that replaces the U-Net in diffusion models for image and video generation.
• Differential privacy: A mathematical privacy guarantee for training data.
• Sparse attention: Scores only a fixed subset of token pairs and sets every other weight to zero in advance.
• Linear variable differential transformer: An electrical sensor that measures linear displacement.