Transformer-XL

Transformer-XL is a transformer language model that carries context across fixed-length text segments by caching the hidden states of the previous segment and letting the next segment attend to them. Zihang Dai, Zhilin Yang, and colleagues at Carnegie Mellon University and Google Brain introduced it in Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context at ACL 2019. A vanilla transformer trained on fixed segments, each a separate context window, cannot see anything before the start of its current segment, a problem the authors called context fragmentation. Transformer-XL learned dependencies 80% longer than recurrent neural networks (RNNs) and 450% longer than vanilla transformers.

Segment-Level Recurrence#

When the model processes a segment, every layer concatenates the cached hidden states of the previous segment with those of the current one. Queries come only from the current segment, while keys and values span both, and gradients stop at the cache, so training cost per segment stays fixed. A single attention layer in PyTorch shows the pattern:

import torch
from torch import nn

torch.manual_seed(0)
seg_len, dim = 4, 16
attn = nn.MultiheadAttention(dim, num_heads=2, batch_first=True)
hidden = torch.randn(1, 3 * seg_len, dim)  # hidden states for three consecutive segments

memory = torch.empty(1, 0, dim)  # nothing is cached before the first segment
for i, segment in enumerate(hidden.split(seg_len, dim=1), start=1):
    # Queries come from the current segment; keys and values also cover the cached one
    context = torch.cat([memory, segment], dim=1)
    out, weights = attn(segment, context, context)
    print(f"Segment {i}: {weights.shape[1]} queries attend over {weights.shape[2]} positions")
    # Cache this segment for the next one; detach() stops gradients as in Transformer-XL
    memory = segment.detach()

The first segment's 4 queries see only that segment's 4 tokens, and every later segment's queries see 8: their own 4 plus the 4 cached ones. Because layer n of a segment reads the cached layer n − 1 states of the segment before, the reach extends one more segment with every layer, about N × L tokens for segment length N and L layers:

The cache also speeds up evaluation. A vanilla transformer predicting token after token has to recompute a full window for every new token, while Transformer-XL reuses the cached states, which made it up to 1,800+ times faster during evaluation. Ultralytics YOLO26 tracking meets the same fragmentation problem when a script passes video to model.track() one frame at a time: persist=True keeps the tracker's state between calls, so each object keeps the same ID from frame to frame.

Relative Positional Encoding#

Reusing cached states breaks absolute positions: the first token of the cached segment and the first token of the current one would carry the same position embedding. Transformer-XL instead computes each attention score from the contents of the query and key plus a sinusoidal encoding of the distance between them, with two learned bias vectors shared across positions, one for content and one for position. Because scores depend only on distances, the model can attend over a longer cache at evaluation than it saw in training. Rotary position embeddings (RoPE), used in most current large language models (LLMs), also make attention scores depend on relative position.

Results and Influence#

Transformer-XL set new state-of-the-art results on five benchmarks: 0.99 bits per character on enwik8, 1.08 on text8, and perplexity of 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank without fine-tuning. Trained only on WikiText-103, it generated reasonably coherent new articles thousands of tokens long. XLNet, from several of the same authors, built its natural language understanding pretraining on the Transformer-XL architecture, and DeepMind's Compressive Transformer extended the cache by compressing old memories instead of discarding them.

Transformer-XL vs. Related Models#

• Long short-term memory (LSTM): Updates one hidden state token by token, so information from far back has to pass through every step in between.
• Longformer: Reads a whole document of 4,096 tokens at once with windowed and global attention, without recurrence.
• KV cache: Stores keys and values of earlier tokens during generation so they are not recomputed. Transformer-XL's memory is also used in training, with gradients stopped at the segment boundary.
• State space models (SSMs): Compress the entire history into a fixed-size state.