Token

A token is the basic unit of input and output that an AI model processes: a word, part of a word, a character, or a byte in text, or a patch of pixels in an image. Before a transformer can read a sentence or an image, a tokenizer splits the input into tokens and maps each one to an integer ID, and the model looks up an embedding vector for every ID. Token counts set both what a model can handle and what it costs: a context window is measured in tokens, large language models (LLMs) generate text one token at a time, and commercial APIs bill per input and output token.

Token vs. Tokenization and Related Terms#

• Tokenization: The process that splits raw input into tokens. A token is one unit of its output.
• Vocabulary and token IDs: A tokenizer's vocabulary is the fixed set of tokens it knows, each with an integer ID. BERT uses 30,522 tokens, GPT-2 50,257, Llama 3 128,256, and Gemma 3 about 262,000.
• Special tokens: Markers for the structure of a sequence, such as start-of-text, end-of-text, and padding tokens, or the [CLS] token whose final embedding BERT uses for classification.
• Embeddings: The learned vectors that the model's first layer looks up for each token ID, which carry the token's meaning into the network.

Text Tokens#

A vocabulary of whole words cannot cover every name, typo, and new term, and single characters make sequences very long. Modern tokenizers therefore use subwords: frequent words become single tokens, and rare words split into reusable pieces. Byte pair encoding (BPE), adapted to neural machine translation by Sennrich and colleagues in 2016, starts from single characters and repeatedly merges the most frequent adjacent pair until the vocabulary reaches its target size. GPT models apply BPE to bytes, so any text, including code and emoji, can be encoded, and WordPiece (used by BERT) and SentencePiece's Unigram model are related subword schemes. CLIP's BPE tokenizer, loaded here from the Hugging Face Hub, shows how words split:

from transformers import AutoTokenizer

# Load CLIP's byte-pair-encoding (BPE) tokenizer from the Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")

for text in ["bus", "double-decker bus", "unhappiness", "Ultralytics"]:
    tokens = tokenizer.tokenize(text)
    print(f"{text!r} -> {tokens} -> {tokenizer.convert_tokens_to_ids(tokens)}")

"bus" is a single token, ID 2840, "unhappiness" splits into un and happiness, and the rare word "Ultralytics" breaks into three pieces, ultr, aly, and tics. The </w> suffix marks the end of a word in CLIP's vocabulary of 49,408 tokens, and CLIP's text encoder reads at most 77 tokens per prompt.

Visual Tokens#

The Vision Transformer (ViT), introduced in the paper An Image Is Worth 16x16 Words, splits an image into fixed-size patches, 16 × 16 pixels in ViT-B/16, and projects each patch into an embedding, so a 224 × 224 image becomes 196 tokens. Position embeddings record where each patch came from, and an extra class token gathers information for classification. The projection is usually implemented as a convolution whose stride equals its kernel size:

import torch

# ViT-B/16 patch embedding: a 16x16 convolution with stride 16 turns each patch into one 768-dim token
patch_embed = torch.nn.Conv2d(3, 768, kernel_size=16, stride=16)

for size in (224, 448):
    image = torch.rand(1, 3, size, size)
    tokens = patch_embed(image).flatten(2).transpose(1, 2)  # (batch, tokens, dims)
    print(f"{size}x{size} image -> {tuple(tokens.shape)}")

A 224 × 224 image yields 196 tokens of 768 values, and doubling the side to 448 pixels yields 784, four times as many. Self-attention compares every token with every other, so four times the tokens means about sixteen times the attention cost, which is why high-resolution models use windowed attention, as in the Swin Transformer, or combine similar tokens with token merging.

Vision language models pass the visual tokens from an image encoder into an LLM alongside text tokens, so one transformer attends over both and writes its answer one text token at a time:

Why Token Counts Matter#

• Context limits: The prompt, any retrieved documents or images, and the generated answer must all fit in the model's context window.
• Cost and latency: APIs price input and output tokens separately, and generation time grows with the number of output tokens, since each one takes a decoding step that reads the model's KV cache.
• Languages and formats: Tokenizers trained mostly on English split other languages, code, and long numbers into more tokens per word, so the same content costs more and fills the context window sooner.
• Character-level tasks: A model sees subword tokens, so tasks that depend on individual letters, such as counting the letters in a word or reversing it, are harder for it than they look.

Real-World Applications#

• Text generation: An LLM assigns a probability to every token in its vocabulary, samples one, appends it to the sequence, and repeats, with sampling methods such as nucleus sampling choosing among the likely tokens.
• Image generation: Autoregressive image models predict discrete visual tokens from a learned codebook, built with vector quantization, the way an LLM predicts words.
• Open-vocabulary detection: YOLOE-26, the YOLO26-based version of Tsinghua University's YOLOE in the Ultralytics package, finds objects from typed class names. Its set_classes() method tokenizes each name with CLIP's BPE tokenizer, Apple's MobileCLIP2 text encoder turns the tokens into an embedding, and the model matches those embeddings against image regions, so it can detect a "double-decker bus", a class outside the COCO dataset, from the words alone.