Constrained Decoding

Constrained decoding is a text generation technique that limits which tokens a language model may produce at each step, so the finished output is guaranteed to match a required format: one of a fixed set of labels, a regular expression, a grammar, or a JSON Schema. The model still scores every token in its vocabulary. Before a token is chosen, the decoder removes every option that would break the format. Applications that parse model output, such as API backends, AI agents, and data-extraction pipelines, use it so that every response parses on the first try.

Masking Invalid Tokens#

A large language model generates text one token at a time. At each step it produces a score, or logit, for every token in its vocabulary, and a softmax turns those scores into probabilities. Constrained decoding adds one operation before sampling: tokens that cannot lead to a valid output get a logit of negative infinity, which gives them zero probability, and the remaining probability is renormalized over the valid tokens.

import torch

# Raw next-token scores from a model over a toy vocabulary
vocab = ["approved", "rejected", "review", "maybe", "{", "}"]
logits = torch.tensor([1.2, 0.4, 0.9, 2.5, 0.1, 0.3])

# The schema allows only the three status labels at this position
allowed = torch.tensor([True, True, True, False, False, False])
masked_logits = logits.masked_fill(~allowed, float("-inf"))

# Softmax renormalizes probability over the valid tokens only
for name, raw, masked in zip(vocab, logits.softmax(-1), masked_logits.softmax(-1)):
    print(f"{name:>9}: {raw:.2f} -> {masked:.2f}")

The unconstrained model puts 56% of its probability on "maybe". With the mask applied, "maybe" drops to zero and the three permitted labels share the probability in proportion to their original scores, so "approved" rises from 15% to 46%.

Production inference engines track a constraint state as generation proceeds. A fixed choice list or a regular expression compiles into a finite-state machine. JSON and other nested formats need a grammar with a stack, because a closing brace is only valid after a matching opening brace. After each token, the decoder advances the state and recomputes the allowed set. Tokenization produces subword pieces of varying length, so a single token such as "}, can cover several grammar steps at once. Libraries therefore precompute which vocabulary entries are legal in each state, which keeps the per-token cost small even for vocabularies of more than 100,000 tokens. Brandon Willard and Rémi Louf described this indexing for regular expressions and context-free grammars in 2023 in Efficient Guided Generation for Large Language Models, the paper behind the Outlines library.

Choosing a Constraint#

The right constraint matches the output the application actually consumes. A choice list suits classification and routing, where the answer must be one of a few strings. Regular expressions cover short structured values such as dates, IDs, and phone numbers. Context-free grammars describe SQL, code, and domain-specific languages, and llama.cpp's GBNF grammar format is a widely used example. For records with named fields, JSON Schema specifies required keys, types, enums, and nesting. Hosted APIs such as OpenAI Structured Outputs accept a schema and enforce it during generation, usually supporting a documented subset of the full JSON Schema specification.

Constrained decoding is often confused with nearby techniques:

• Structured outputs: The broader goal of machine-readable responses, reached either by constrained decoding at generation time or by validating the output afterward and retrying on failure.
• JSON mode: Constrains output to syntactically valid JSON of any shape, so keys, types, and values still need checking.
• Prompt engineering: Requests a format in the instructions, which models follow most of the time. Schema-constrained requests usually describe the fields in the prompt as well, so the model knows what each one means.
• Function calling: Tool names and arguments are among the most common targets for schema constraints, since the calling code expects exact field names.

Trade-offs in Practice#

A guaranteed format guarantees nothing about content. A schema can force {"total": 1290.00} to be a number, but the number can still be wrong, and a constrained model can hallucinate fluently inside valid fields. Applications still validate business rules after generation.

Constraints can also lower answer quality when they push a model away from the text it would naturally write. A schema that demands the final answer as its first field leaves no room for intermediate reasoning, so schemas often place a free-text reasoning or notes field before the answer field. Output length needs attention too: if generation hits the token limit in the middle of an object, the result is invalid regardless of the constraint, which is why structured requests set maximum lengths with headroom for the full structure. Very large or deeply recursive schemas take longer to compile, which matters for systems that build a new schema for each request.

Vision applications use the same mechanism when a multimodal model must return fields about an image. The Ultralytics LLM interface forwards request arguments unchanged to OpenAI-compatible APIs, so a JSON Schema passed with an image request limits a vision-language model's answer to fields such as hazard and severity, ready to combine with YOLO26 detections from the same frame.