Nucleus Sampling

Nucleus sampling, also called top-p sampling, is a way to choose the next token during AI text generation. Instead of always picking the most likely token, the model builds a shortlist whose probabilities add up to at least a chosen threshold, then randomly samples from that shortlist. The shortlist changes with each prediction: it can be small when one continuation is obvious and larger when several continuations are plausible.

How Nucleus Sampling Works#

A language model assigns a probability to every possible next token. These probabilities are commonly calculated from the model’s output scores using softmax, which makes the values add up to one. Nucleus sampling sorts the tokens from most to least likely, includes them until their cumulative probability reaches the threshold p, excludes the rest, and samples from the included tokens. Here, p stands for probability mass, not the probability of one individual token. PyTorch’s softmax documentation explains the probability conversion, while IBM’s decoding guide describes the cumulative cutoff. (docs.pytorch.org)

Suppose four possible next tokens have probabilities of 0.50, 0.25, 0.15, and 0.10. With top_p=0.80, the first two total only 0.75, so the third is included too, bringing the shortlist to 0.90. The fourth is excluded. The model then samples among the first three in proportion to their probabilities after normalization; it does not give each an equal chance. The cutoff may exceed p because the token that crosses it remains in the shortlist. Google Cloud’s explanation of top-p describes the same token-selection rule. (cloud.google.com)

This selection happens again after every generated token. As the sentence develops, the model computes a new distribution and therefore a new nucleus.

Top-p vs. Top-k, Greedy Decoding, and Temperature#

These settings affect different parts of generation:

• Top-k sampling keeps a fixed number of candidates, such as the five most likely tokens. Top-p keeps enough candidates to reach a probability threshold, so its shortlist has no fixed size.
• Greedy decoding always selects the single most likely token. It is predictable but offers none of the variation that sampling allows.
• Temperature changes how strongly the model favors high-probability tokens before selection. Lower temperatures concentrate probability on leading choices; higher temperatures spread it more widely. Top-p instead sets a cumulative cutoff on the resulting distribution.

Implementations can combine these controls, but their interaction makes results harder to interpret. When experimenting, change one setting at a time. The OpenAI Chat Completions parameter reference describes top_p and recommends adjusting either it or temperature rather than both at once. (platform.openai.com)

Where It Matters in Practice#

In a customer-support chatbot, nucleus sampling can allow several natural phrasings of a routine answer without drawing from every unlikely continuation. For example, an assistant explaining a return policy might vary its opening sentence while preserving the policy details supplied in its prompt. Sampling does not verify those details: a fluent answer can still be wrong, so factual checks and appropriate grounding remain necessary.

In image captioning, a vision-language model may have several reasonable ways to describe the same street scene. Top-p can vary the wording of captions while filtering out low-probability token choices. A visual pipeline might first use Ultralytics YOLO26 for object detection, then provide detected objects as context to a caption generator. YOLO’s detection step does not use nucleus sampling to choose its object labels; the setting applies to the downstream language-generation step. The TensorFlow image-captioning tutorial illustrates how visual input and a text decoder fit together. (ibm.com)

Trying Top-p in a Documented Workflow#

The Ultralytics LLM interface accepts request arguments for a compatible language-model endpoint. After installing ultralytics[llm] and setting OPENAI_API_KEY, this example requests a short response with top_p=0.9:

from ultralytics import LLM

# Use an endpoint that supports the top_p request argument.
llm = LLM("gpt-4.1-mini", api="chat.completions")

prompt = "Describe a city bus in one friendly sentence."
response = llm(prompt, top_p=0.9)

message = response.choices[0].message
print(message.content)

The code leaves token selection to the language-model provider. Running it again may produce different wording, but top_p=0.9 is not a promise that every response will differ. Check the selected model’s supported parameters before applying the same setting elsewhere.

Choosing a Useful Setting#

Start with the model’s default, then compare outputs on representative prompts. Lower top_p if responses wander into unlikely wording; consider a higher value if they are needlessly repetitive. Judge the result by the task—not by variety alone. For customer answers or captions, review factual accuracy and whether important details are present. A narrower shortlist can reduce some unusual continuations, but it cannot make unsupported statements true. IBM’s guidance on generating accurate output likewise treats decoding choices as only one part of a grounded generation workflow. (ibm.com)