Tokenization

Tokenization is the step that converts raw input into a sequence of tokens, the discrete units a model reads, and gives each token an integer ID from a fixed vocabulary. A text tokenizer turns "The bus stopped." into four IDs, and the model looks up an embedding vector for each one, so the tokenizer decides what a large language model (LLM) treats as a single unit. Tokens are also the unit of a model's context window, of inference latency during text generation, and of API pricing. The same idea extends beyond text: a Vision Transformer (ViT) tokenizes an image into patches, and speech models tokenize audio into short frames.

How Subword Tokenization Works#

Early natural language processing (NLP) split text on spaces and punctuation, which leaves rare and new words outside the vocabulary, while character-level tokenization makes sequences very long. Modern transformers use subword tokenizers that keep frequent words whole and break rare ones into reusable pieces. Byte pair encoding (BPE), adapted for machine translation by Sennrich et al. in 2015, learns its vocabulary from a training corpus: it starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size. Encoding new text replays the learned merges in order.

Byte-level BPE, used since GPT-2, starts from the 256 possible byte values, so any string, emoji, or source file can be encoded without an unknown token. Other algorithms choose their pieces differently. WordPiece, used by BERT, merges the pair that most increases the likelihood of the training data, and the Unigram method in Google's SentencePiece library starts from a large candidate vocabulary and prunes it, treating the space as an ordinary symbol so one tokenizer handles languages written without spaces. Vocabularies have grown over time: BERT uses 30,522 tokens, GPT-2 and GPT-3 use 50,257, Llama 3 uses 128,256, and OpenAI's o200k_base uses about 200,000. OpenAI's tiktoken library compares the oldest and newest of those vocabularies:

import tiktoken

# Byte-level BPE vocabularies from GPT-2 (2019) and GPT-4o (2024)
for name in ["gpt2", "o200k_base"]:
    encoding = tiktoken.get_encoding(name)
    print(f"{name}: {encoding.n_vocab:,} tokens in the vocabulary")
    for text in ["Ultralytics YOLO26 detects buses.", "バスが止まった。"]:
        ids = encoding.encode(text)

        # Decode each ID on its own to see where the tokenizer split the text
        pieces = [encoding.decode([i]) for i in ids]
        print(f"  {len(ids)} tokens: {pieces}")

GPT-2's vocabulary spends 11 tokens on the English sentence, splitting "Ultralytics" into Ult, ral, y, and tics, while o200k_base needs 9. For the Japanese sentence "the bus stopped," GPT-2 needs 9 tokens and splits the character 止 into two byte-level tokens that each decode alone as the replacement character �, while o200k_base needs 7.

Tokenization vs. Related Terms#

• Token: The unit that tokenization produces, such as a word, a subword, a byte, an image patch, or an audio frame.
• Embeddings: The learned vectors a model looks up for each token ID, trained with the rest of the network over a vocabulary the tokenizer fixed beforehand.
• Word segmentation in classical NLP: Libraries such as spaCy and NLTK split text into words and punctuation for parsing and tagging, often followed by stemming or lemmatization for linguistic analysis.
• Tokenization in data security: In payments and data protection, tokenization replaces a sensitive value, such as a card number, with a random surrogate stored in a protected vault.

Why Token Counts Matter#

For English text, OpenAI's tokenizers average about four characters per token, but other languages and code can need far more. Petrov et al. (2023) found that the same text translated into different languages can differ in tokenized length by up to 15 times, which raises API costs and shrinks the usable context for those languages. Token boundaries also shape what a model can see. An LLM that reads "strawberry" as a few multi-letter tokens has no direct view of its letters, and o200k_base splits "3.14159" into 3, ., 141, and 59, which makes digit-level arithmetic harder. Tokenizer-free designs such as Meta's Byte Latent Transformer avoid a fixed vocabulary by grouping raw bytes into patches whose size depends on how predictable the next byte is.

Tokenizing Images, Audio, and Video#

A ViT cuts an image into fixed-size patches, so with 16×16-pixel patches a 224×224 image becomes 196 patch tokens, each flattened and linearly projected into an embedding before self-attention. Vision-language models (VLMs) pass patch embeddings to an LLM alongside text tokens, so a higher image resolution means more tokens per request. Image generators often use discrete visual tokens instead: a vector quantization codebook replaces each patch with the ID of its nearest code vector, which lets visual autoregressive models generate images token by token like text, and token merging reduces the count for long image and video sequences by combining similar tokens.

Vision models tokenize text as well. When YOLOE-26 receives class names through set_classes, Ultralytics runs each name through CLIP's byte-level BPE tokenizer, padded or truncated to 77 tokens, and Apple's MobileCLIP2 text encoder turns the result into the embedding that the detection head matches against image regions. In speech recognition, OpenAI's Whisper converts each 30-second audio window into log-Mel spectrogram frames, which its encoder processes as a sequence of 1,500 positions.