Text Generation

Text generation is the automatic production of written language by a model, from a one-line reply to a full report or program. Today it is mostly done by large language models (LLMs) that work autoregressively: given a prompt, the model computes a probability for every token in its vocabulary, picks one, appends it, and repeats until it decides to stop. The same loop drives chat assistants, code completion, machine translation, and text summarization. Earlier systems filled templates or sampled from n-gram and recurrent neural network models, and the transformer made long, coherent outputs practical from GPT-2 in 2019 onward.

How Autoregressive Generation Works#

Tokenization turns the prompt into token IDs, and the model processes all of them in one parallel prefill pass, storing each token's attention keys and values in a KV cache. Every decode step then runs only the newest token through the model, reads the cache, and produces logits, one raw score per vocabulary entry, that a softmax turns into probabilities. A decoding strategy picks the next token, which streams to the user and joins the sequence. Generation stops at an end-of-sequence token, a stop string, or a token limit, so output length is decided during generation:

Prefill is compute-heavy and decode is limited by memory bandwidth, so serving engines schedule the two carefully, for example with chunked prefill. Streaming is how applications show text as it appears: the Ultralytics LLM interface passes stream=True through to the OpenAI SDK, so a YOLO26 application can print a model's description of a detected scene token by token as each response.output_text.delta event arrives.

Decoding Strategies#

The decoding strategy shapes the output as much as the model does. Greedy decoding always takes the most likely token, which is deterministic but tends to repeat itself. Beam search keeps the k most likely partial sequences and returns the best complete one, which suits tasks with one right answer, such as translation. Sampling draws from the distribution, usually after reshaping it: temperature divides the logits before the softmax, top-k sampling keeps only the k most likely tokens, and nucleus (top-p) sampling, proposed in The Curious Case of Neural Text Degeneration by Holtzman et al. in 2019, keeps the smallest set of tokens whose probabilities add up to p. A five-token vocabulary in PyTorch shows both effects:

import torch

# Next-token logits over a five-word vocabulary after the prompt "I took the"
vocab = ["bus", "train", "car", "stairs", "banana"]
logits = torch.tensor([3.0, 2.4, 1.8, 0.6, -2.0])

# Temperature divides the logits: low values sharpen the distribution, high values flatten it
for temperature in [0.5, 1.0, 1.5]:
    probs = torch.softmax(logits / temperature, dim=-1)
    print(f"T={temperature}:", ", ".join(f"{w} {p:.2f}" for w, p in zip(vocab, probs)))

# Top-p (nucleus) sampling keeps the smallest set of tokens whose probabilities reach p
probs, order = torch.softmax(logits, dim=-1).sort(descending=True)
keep = probs.cumsum(dim=-1) - probs < 0.9
print("Top-p 0.9 keeps:", [vocab[i] for i in order[keep]])

# Sample from the renormalized nucleus
torch.manual_seed(0)
picks = torch.multinomial(probs[keep] / probs[keep].sum(), num_samples=10, replacement=True)
print("Samples:", [vocab[order[i]] for i in picks])

At temperature 0.5, "bus" gets probability 0.71; at 1.5 it drops to 0.42 and "banana" rises from 0.00 to 0.02. Top-p 0.9 keeps "bus," "train," and "car," which together hold 0.95 of the probability, so "stairs" and "banana" can never be sampled, and ten draws produce a varied mix of the three survivors. Other controls act on the same logits: repetition penalties lower the scores of tokens already used, and constrained decoding masks every token that would break a grammar or JSON schema, which is how APIs guarantee structured outputs. Speculative decoding speeds generation by letting a small draft model propose several tokens that the large model verifies in one pass.

Text Generation vs. Related Terms#

• Language modeling: Learning the probability of the next token, the training objective that makes generation possible. Generation is the inference-time use of that model.
• Sequence-to-sequence models: Encoder-decoder models that generate an output text from a separate input, as in translation. Decoder-only LLMs handle the same tasks by continuing a prompt.
• Diffusion language models: Generate a whole passage by refining all positions in parallel over several denoising steps instead of one token at a time.
• Text-to-image: Takes text as input and produces pixels.

Quality and Risks#

Generated text is fluent whether or not it is correct, so hallucinations are the main risk in factual work, and retrieval-augmented generation (RAG) reduces them by placing source documents in the prompt. Low temperatures make answers more repeatable, and high temperatures add variety along with a higher chance of picking an unlikely, wrong token. Machine-written text is also hard to spot, which drives work on AI watermarking. In the 2020 GPT-3 paper, people asked to tell 200-word news articles written by the 175-billion-parameter model from human-written ones were right only 52% of the time.