Structured Outputs

Structured outputs are model responses constrained to follow a predefined format, usually a JSON Schema that names the required keys, their types, and the allowed values, so that software can parse every reply without cleanup. A large language model (LLM) asked politely for JSON usually complies, but it can still add a sentence of prose, misspell a key, or stop mid-object, and each of these breaks the program reading the reply. Structured outputs remove that risk with constrained decoding: at every step of generation, the inference engine allows only the tokens that keep the output valid against the schema. When OpenAI added the feature to its API in August 2024, it reported that gpt-4o-2024-08-06 with Structured Outputs scored 100% on its evaluation of complex JSON schema following, compared with under 40% for GPT-4 (gpt-4-0613) without it.

JSON Mode vs. Structured Outputs#

Provider APIs offer two levels of guarantee. JSON mode, which OpenAI introduced in November 2023, guarantees only that the reply is syntactically valid JSON, so keys can still be missing or misnamed. Structured outputs guarantee that the reply matches the supplied schema. In OpenAI's Structured Outputs API, developers define the schema as a Pydantic model in Python or a Zod object in JavaScript, and a refusal arrives in a separate field instead of breaking the format. Anthropic's Claude structured outputs compile each schema into a grammar, which adds latency the first time a schema is used, and cache the compiled grammar for 24 hours. Both providers support only a subset of JSON Schema; Anthropic's, for example, does not support recursive schemas or numeric limits such as minimum.

How Constrained Decoding Works#

At each step, an LLM produces a score, or logit, for every token in its vocabulary, and decoding picks from the resulting softmax distribution. Constrained decoding sets the score of every token that would break the format to minus infinity first, so those tokens can never be chosen. Brandon Willard and Rémi Louf showed how to do this efficiently in Efficient Guided Generation for Large Language Models: a regular expression or JSON Schema is compiled into a finite-state machine, and an index built once over the vocabulary lists the valid tokens for each state, so each step costs a lookup. Their open-source Outlines library implements the method, and serving engines such as vLLM offer grammar-guided generation for open models.

The following example applies the idea by hand to GPT-2 from Hugging Face Transformers. The schema's fixed text, the braces, keys, and quotes, is written by code, and at each value the model may choose only among the tokens the schema allows:

import json

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
prompt = "Review: The battery died after two days and support never replied.\nJSON:"

# Unconstrained greedy decoding: nothing forces the reply into a schema
ids = tokenizer(prompt, return_tensors="pt").input_ids
free = model.generate(ids, max_new_tokens=16, do_sample=False, pad_token_id=tokenizer.eos_token_id)
print("free:", repr(tokenizer.decode(free[0, ids.shape[1] :])))


def choose(text, options):
    """Return the option GPT-2 scores highest after text, with every other token masked out."""
    logits = model(tokenizer(text, return_tensors="pt").input_ids).logits[0, -1]
    allowed = torch.tensor([tokenizer.encode(option)[0] for option in options])
    masked = torch.full_like(logits, float("-inf"))
    masked[allowed] = logits[allowed]
    return tokenizer.decode(masked.argmax())


# Constrained decoding: the schema writes the keys, and the model may only pick allowed values
with torch.no_grad():
    text = prompt + ' {"sentiment": "'
    text += choose(text, ["positive", "negative"]) + '", "stars": '
    text += choose(text, ["1", "2", "3", "4", "5"]) + "}"
print("constrained:", json.loads(text[len(prompt) :]))

It prints free: ' The battery died after two days and support never replied.\n\nJSON: The' and constrained: {'sentiment': 'positive', 'stars': 1}. Left alone, GPT-2 repeats the review instead of writing JSON, while the constrained version always parses. The parsed result also shows the limit of the technique: the 124-million-parameter model judged a dead battery and silent support as positive, with one star. The schema fixed the format, and the values inside it still came from the model's own judgment.

Uses and Trade-Offs#

• Data extraction: Pulling names, dates, totals, and line items from invoices, contracts, or emails directly into database rows.
• Function calling: A tool call is a structured output whose schema is the tool's parameters. Strict tool use in the OpenAI and Anthropic APIs applies the same guarantee to tool arguments.
• Agentic workflows: Routing decisions as enum values, and intermediate results passed between the steps of a prompt chain as typed objects.
• Vision-language models: Detections returned as lists of labels and box coordinates that drawing or evaluation code can consume directly.

Class-prompted smart annotation in Ultralytics Platform depends on that last kind of output. It asks vision models from providers such as OpenAI, Anthropic, and Google to detect a dataset's classes and adds every returned object as an editable bounding box, so each reply must name a class and give coordinates in a form code can parse. These models return no confidence scores, so Platform hides the confidence and IoU settings while one is selected.

Strict formats can cost accuracy on reasoning tasks. In Let Me Speak Freely?, Zhi Rui Tam and colleagues found a significant decline in LLM reasoning under format restrictions, with stricter constraints causing larger drops. OpenAI's API generates keys in the order the schema lists them, so a common remedy is a free-text reasoning field placed before the answer field, so the model writes its chain of thought before committing to the constrained answer.