Perplexity

Perplexity is a metric for how well a probabilistic model predicts a sequence, computed as the exponential of the average negative log-likelihood the model assigns to each actual next token. Lower values are better. A perplexity of 10 means the model is, on average, as uncertain as if it were choosing uniformly among 10 options at every step, which is why the number is often read as an effective branching factor. Frederick Jelinek, Robert Mercer, Lalit Bahl, and James Baker introduced it at IBM in 1977 to measure the difficulty of speech recognition tasks, and it remains the standard intrinsic metric for language modeling. OpenAI's 1.5-billion-parameter GPT-2, for example, reached a perplexity of 17.48 on the WikiText-103 benchmark in 2019 without training on it.

The word also names two unrelated things. Perplexity AI is an AI search engine company, and t-SNE has a perplexity hyperparameter that sets roughly how many neighbors each point considers when t-SNE builds a 2D map of high-dimensional data.

Computing Perplexity#

For a sequence of N tokens, perplexity = exp(-(1/N) x sum of log p(token_i | all previous tokens)). The quantity inside the exponential is the cross-entropy between the text and the model's predictions, the same loss function used to train large language models (LLMs). With natural logarithms the base is e; with base-2 logarithms it is 2, and both give the same perplexity. PyTorch's cross-entropy loss uses natural logarithms and averages over tokens by default, so a language model's perplexity is simply exp(loss).

import math

# Probability the model assigned to each correct next token
token_probs = [0.50, 0.25, 0.80, 0.40]

# Cross-entropy is the average negative log-likelihood, in nats
cross_entropy = -sum(math.log(p) for p in token_probs) / len(token_probs)
print(f"Cross-entropy: {cross_entropy:.3f}, perplexity: {math.exp(cross_entropy):.2f}")

# A model that spreads probability evenly over 10 tokens has perplexity 10
uniform = -sum(math.log(0.1) for _ in token_probs) / len(token_probs)
print(f"Uniform perplexity: {math.exp(uniform):.2f}")

The four probabilities give a cross-entropy of 0.805 nats and a perplexity of 2.24, so the model behaves as if it were choosing among about two equally likely tokens at each step. The uniform case confirms the branching-factor reading: probability 0.1 on every correct token gives exactly 10.

Long documents need care. A model with a fixed context window scores text in chunks, and tokens at the start of a non-overlapping chunk see almost no context, which inflates perplexity. A sliding-window evaluation overlaps the chunks so every token gets as much preceding context as the model allows, at the cost of more computation.

Comparing Models Fairly#

Perplexity is comparable only between models evaluated on the same text with the same tokenization. A model with a larger vocabulary splits text into fewer, harder-to-predict tokens, so its per-token perplexity is not directly comparable with a model that uses smaller tokens. Normalizing the total log-likelihood by the number of bytes or characters, reported as bits per byte or bits per character, removes that dependence.

The test data must also be absent from the training data. A model that has seen a passage before assigns it unusually high probability, so this form of data leakage makes perplexity look better than real performance. The same effect drives membership inference attacks, which flag text with suspiciously low perplexity as likely training data.

Perplexity scores next-token prediction on a fixed text. Chat models go through instruction tuning and reinforcement learning from human feedback (RLHF) to follow requests and match human preferences, so they are compared with task benchmarks and human preference ratings.

Uses Beyond Model Comparison#

Data curation is a major use. The CCNet pipeline, released by Facebook AI researchers in 2019, scored Common Crawl pages with a language model trained on Wikipedia and split them by perplexity into head, middle, and tail thirds, with the low-perplexity head treated as the highest-quality text. Many pretraining pipelines filter web data in a similar way, a step of data cleaning at web scale. AI-text detectors such as GPTZero also score perplexity, since text sampled from a language model tends to be more predictable than human writing.

The formula applies to any model trained with cross-entropy. Ultralytics YOLO26 classification models are trained with mean cross-entropy loss, so exp(validation loss) is the perplexity over the true class labels. A validation loss of 0.35 corresponds to a perplexity of about 1.42, meaning that on an average image the model spreads its confidence as if hesitating between roughly 1.4 classes.