Nucleus Sampling

Nucleus sampling, also called top-p sampling, is a decoding method that picks each next token from the smallest set of candidates whose combined probability reaches a threshold p, then samples from that set. Ari Holtzman and colleagues at the University of Washington and the Allen Institute for AI introduced it in 2019 in The Curious Case of Neural Text Degeneration. Working with GPT-2 Large, they showed that greedy decoding and beam search make language models fall into bland, repetitive loops, while sampling from the full distribution drifts into incoherent text, and that cutting off the unreliable tail of the distribution produced text whose perplexity and variety came closest to human writing. Top-p is now a standard setting in text generation APIs for nearly every large language model.

How the Nucleus Is Chosen#

At each step, the language model scores every token in its vocabulary, and softmax turns those scores into probabilities. Nucleus sampling then sorts the tokens from most to least likely, adds up their probabilities in that order until the running total reaches p, discards everything outside that set, renormalizes the survivors so they sum to one, and samples a single token.

Take four candidates with probabilities 0.50, 0.25, 0.15, and 0.10 and set p to 0.8. The first two sum to 0.75, short of the threshold, so the third joins and the nucleus covers 0.90. The fourth token is dropped, and the three survivors are drawn with probabilities of roughly 0.56, 0.28, and 0.17 after renormalization.

The defining property is that the nucleus resizes itself at every step. After "The capital of France is," one token holds nearly all the probability and the nucleus shrinks to a single word. At the start of a story, probability spreads across hundreds of plausible words and the nucleus widens to include them. A fixed number of candidates cannot adapt this way.

The same procedure in PyTorch, for one decoding step over a six-token vocabulary:

import torch

# Next-token probabilities for a 6-token vocabulary
probs = torch.tensor([0.10, 0.40, 0.04, 0.25, 0.06, 0.15])
top_p = 0.75

# Sort, then keep tokens until the running total reaches top_p
sorted_probs, order = probs.sort(descending=True)
cumulative = sorted_probs.cumsum(dim=0)
keep = cumulative - sorted_probs < top_p  # the token that crosses top_p stays

# Renormalize the nucleus and sample the next token id
nucleus = sorted_probs * keep
next_token = order[torch.multinomial(nucleus / nucleus.sum(), num_samples=1)]
print(order[keep].tolist(), next_token.item())

The output lists the nucleus, token ids 1, 3, and 5, which together hold 0.80 of the probability, followed by the sampled id. Tokens 0, 2, and 4 can never be chosen at this step, and rerunning the script picks among the three survivors in proportion to their probabilities.

Top-p vs. Greedy Decoding, Top-k, and Temperature#

• Greedy decoding always takes the single most likely token. It is deterministic and works for short factual answers, but long outputs tend to loop.
• Top-k sampling keeps a fixed number of candidates. With k set to 50, a confident step still samples among 50 tokens, including nonsense, while an uncertain step may cut off reasonable options. Top-p replaces the fixed count with a fixed share of probability.
• Temperature divides the model's scores before softmax. Values below 1 sharpen the distribution toward the top choices, and values above 1 flatten it. Because top-p is applied after temperature, raising the temperature also widens the nucleus.
• Min-p sampling keeps every token whose probability is at least a set fraction of the top token's probability, so its cutoff also tracks the model's confidence. Hugging Face Transformers, vLLM, and llama.cpp offer it alongside top-p.

Since temperature and top-p both control randomness, the standard advice is to adjust one or the other rather than both at once. Many inference engines also let top-k and top-p run together, with top-k acting as a hard cap on the candidate count.

Choosing a Top-p Value#

A top-p of 1.0 disables the filter, and common defaults fall between 0.9 and 1.0. Top-p only reshapes the model's own probabilities, so factual accuracy comes from grounding the prompt, for example with retrieval-augmented generation, which reduces hallucinations by giving the model the facts it needs. Higher values add variety for brainstorming, chatbot dialog, and creative writing. Lower values such as 0.5 make outputs more predictable and repetitive, which suits data extraction, classification prompts, and code, and constrained decoding goes further by masking every token that would break a required format.

The same setting governs language output in vision pipelines. When an Ultralytics YOLO26 detection triggers a scene description from a vision-language model, the Ultralytics LLM interface passes request arguments such as top_p straight through to the provider's API, so a low value such as 0.5 keeps the descriptions of similar frames worded alike.