Transformer

A transformer is a neural network architecture that processes a sequence of tokens, such as words or image patches, by letting every token draw information from every other token through self-attention. Google researchers introduced it in 2017 in Attention Is All You Need for machine translation. Earlier recurrent neural networks (RNNs) read a sentence one word at a time, so training could not be parallelized across positions. The transformer processes all positions at once, and its base model reached state-of-the-art English-to-German translation after 12 hours of training on eight GPUs. Transformers now underlie large language models (LLMs), vision transformers, speech recognition, and many object detection and image generation models.

How Transformers Work#

Input tokens are first converted into embeddings, vectors of a fixed width (512 in the original model). A transformer then applies a stack of identical layers, each with two sublayers:

• Multi-head self-attention: Each token produces a query, a key, and a value vector. A token's output is a weighted average of all value vectors, weighted by how well its query matches each key, as described in the attention mechanism. Several heads run in parallel, 8 in the original model, so different heads can track different relationships.
• Feed-forward network: A two-layer network applied to each token separately, which transforms the information that attention gathered.

A residual connection wraps each sublayer and layer normalization keeps activations stable. The original model stacked 6 such layers in an encoder, which reads the source sentence, and 6 in a decoder, which writes the translation one token at a time. Each decoder layer adds a masked self-attention step that hides future tokens and a cross-attention step that reads the encoder's output:

Attention compares tokens by content alone, so a transformer layer has no built-in sense of order: shuffling the input tokens only shuffles the outputs. Position information is therefore added to the token embeddings, through fixed sine and cosine patterns in the original paper, learned position vectors in BERT and ViT, or rotary position embeddings (RoPE) in most current LLMs. This PyTorch code shows the effect with one encoder layer:

import torch
from torch import nn

# One transformer encoder layer and a sequence of 5 token embeddings
torch.manual_seed(0)
layer = nn.TransformerEncoderLayer(d_model=32, nhead=4, batch_first=True).eval()
tokens = torch.randn(1, 5, 32)
reverse = torch.arange(4, -1, -1)

# Self-attention alone ignores order: reversing the input only reverses the output
print(torch.allclose(layer(tokens[:, reverse])[:, reverse], layer(tokens), atol=1e-5))

# Adding one learned vector per position makes each output depend on where its token sits
positions = nn.Embedding(5, 32).weight
reordered = layer(tokens[:, reverse] + positions)[:, reverse]
print(torch.allclose(reordered, layer(tokens + positions), atol=1e-5))

It prints True and then False: reversing the tokens only reverses the outputs until position vectors are added, after which the order of the tokens changes every token's output.

Encoder, Decoder, and Encoder-Decoder Models#

The three ways of using the original stack serve different jobs:

• Encoder-only: Every token attends to the whole input in both directions, which suits understanding tasks. BERT for text and ViT for images are encoder-only.
• Decoder-only: Each token attends only to the tokens before it, so the model generates text one token at a time. GPT models and most current LLMs use this design.
• Encoder-decoder: An encoder reads one sequence and a decoder writes another while attending to it, as in translation models, T5, and the Whisper speech recognizer.

Transformers in Computer Vision#

The Vision Transformer (ViT) treats an image as a sequence of 16×16-pixel patches and applies an unmodified transformer encoder, and the Swin Transformer limits attention to local windows to scale to high-resolution images. DETR reframed detection as set prediction: a transformer decoder turns a fixed set of learned object queries into boxes, one object per query, which removes non-maximum suppression (NMS) and hand-designed anchors. The Ultralytics package runs Baidu's RT-DETR, a real-time version of this design whose hybrid encoder applies a transformer encoder layer only to the stride-32 feature map, 400 tokens for a 640×640 image, and whose 6-layer decoder refines 300 object queries into final detections without NMS. Transformers also generate images: diffusion transformers have replaced the U-Net in recent text-to-image and video models, and vision language models feed image tokens into a decoder-only LLM.

Transformers vs. Related Architectures#

• Convolutional neural networks (CNNs): A convolution sees a small neighborhood with the same filter at every position, a built-in assumption that helps on small datasets. Attention can link any two positions within one layer but has to learn spatial structure from data, so transformers usually need more training data or pretraining to beat CNNs.
• RNNs and LSTMs: They carry a fixed-size state from step to step, so memory per token stays constant but training is sequential. A transformer keeps every past token available and trains in parallel.
• State space models: Architectures such as Mamba process sequences with a recurrent state whose cost grows linearly with length, and hybrid models interleave them with attention layers.

Self-attention compares every token with every other, so its cost grows with the square of the sequence length. A 4,096-token sequence has about 16.8 million query-key pairs in every head of every layer, and doubling the length to 8,192 tokens quadruples that to about 67 million. FlashAttention reduces the memory traffic of this computation, and the KV cache stores the keys and values of earlier tokens so that a decoder computes them only once during generation.