Rotary Position Embedding (RoPE)

Rotary Position Embedding (RoPE) gives a transformer information about token order by rotating each query and key vector through an angle proportional to the token's position. Self-attention on its own cannot tell one word order from another, so models must inject position somehow, and with RoPE the attention score between a query at position m and a key at position n depends only on their offset m − n. Introduced in 2021 with the RoFormer model and popularized by EleutherAI's GPT-J and GPT-NeoX, RoPE is now the standard position encoding of open large language models (LLMs), including Llama, Mistral, Qwen, Gemma, and DeepSeek.

How RoPE Works#

RoPE splits each query and key vector of dimension d into d/2 pairs of features and treats each pair as a point in a plane. At position m, pair i is rotated by the angle m·θᵢ, with θᵢ = base^(−2i/d) and a base of 10,000 in the original paper. The first pairs rotate quickly and resolve fine local order, while the last pairs rotate slowly and still separate tokens thousands of positions apart. Rotating a query by mθ and a key by nθ leaves their dot product dependent only on the difference (m − n)θ, and rotation never changes a vector's length. The rotation is applied inside every attention layer, after the query and key projections; value vectors are not rotated, and nothing is added to the token embeddings at the input:

The snippet below implements the rotation with complex numbers, as Meta's Llama reference code does, and checks the relative-position property:

import torch

torch.manual_seed(0)
dim = 64
freqs = 10000 ** (-torch.arange(0, dim, 2) / dim)  # one rotation frequency per pair of features


def rope(x, pos):
    """Rotate each consecutive feature pair of x by the angle pos * frequency."""
    pairs = torch.view_as_complex(x.reshape(-1, 2))
    return torch.view_as_real(pairs * torch.polar(torch.ones_like(freqs), pos * freqs)).flatten()


q, k = torch.randn(2, dim)
# The same query and key, three tokens apart, at two different places in the sequence
print(f"positions 5 and 2:     {rope(q, 5) @ rope(k, 2):.4f}")
print(f"positions 105 and 102: {rope(q, 105) @ rope(k, 102):.4f}")
print(f"positions 5 and 4:     {rope(q, 5) @ rope(k, 4):.4f}")  # one token apart
print(f"Vector length kept: {rope(q, 5).norm():.4f} vs {q.norm():.4f}")

It prints -8.2811 for both pairs three tokens apart, whether they sit at positions 5 and 2 or 105 and 102, and -11.4563 when the same vectors are one token apart. The rotated query keeps its length of 8.3705.

RoPE vs. Other Position Encodings#

• Sinusoidal absolute encoding: The original transformer adds fixed sine and cosine vectors to the input embeddings once, before the first layer. Relative distance has to be inferred from the mixed-in signal.
• Learned absolute embeddings: BERT and GPT-2 learn one vector per position, up to 512 and 1,024 positions, and have no embedding for any position past that limit.
• Relative position biases: T5 and ALiBi add a learned or fixed distance-based bias to the attention scores. RoPE encodes relative distance through the rotation itself and adds no parameters.

Extending the Context Window#

A model trained with RoPE on sequences of length L has never seen the rotation angles of positions beyond L, and attention degrades when it meets them. Context extension methods rescale the angles instead of retraining from scratch. Position Interpolation squeezes new positions into the trained range and extended Llama models to a 32,768-token context window within 1,000 fine-tuning steps. NTK-aware scaling and YaRN stretch the slow, low-frequency pairs more than the fast ones, and YaRN extended Llama 2 to 128K tokens with 10 times fewer training tokens than earlier methods. Llama 3 raised the base from 10,000 to 500,000 during pretraining, so the slowest pairs rotate less per token and distant positions stay distinguishable.

RoPE in Vision Models#

Images need two coordinates, so Vision Transformers use 2D, or axial, RoPE: half of each head's feature pairs rotate with a patch's column and half with its row. Qwen2-VL adds a third, temporal axis for video in its multimodal M-RoPE. Meta's SAM 2, which runs in the Ultralytics package, applies axial RoPE in the four layers of its memory attention, where features of the current video frame attend to stored features of earlier frames. The keys of every stored frame get the same grid of angles as the current frame's queries, so attention between the two depends on the spatial offset between patches.