Structured Output: Getting Reliable JSON from an LLM
On this page
An expense app asks a model to read receipts and "reply in JSON". It works in the demo. In production, one reply in a few hundred starts with "Sure! Here is the data:", another writes the total as "about 12.50", another says the date is "yesterday", and the parser either crashes or, worse, stores the wrong type. Software needs data with a guaranteed shape, and a model left to itself only produces text that usually looks right.
How it works

Three levels of guarantee

- Prompt only. Describe the format and give an example. Works most of the time with good models; nothing guarantees it.
- JSON mode or tool calling. Many APIs can force the reply to be valid JSON, or return arguments for a "tool" you declared. The syntax is guaranteed; fields and types may still not match what you wanted.
- Schema-constrained output. You pass a JSON Schema (field names, types, required fields, allowed values) and the provider or serving engine makes the model follow it. The output is valid and matches the schema.
How constrained decoding works
The model produces one token at a time. With a schema, the decoder knows at each step which tokens could still lead to a valid document, and sets the probability of every other token to zero before sampling.

{"total": the schema expects a number, so only digit tokens survive. The model can't write "about".Self-hosted serving engines offer the same through grammar-based decoding (JSON Schema, regular expressions or context-free grammars), with libraries that turn a schema into a token mask efficiently.
Validate anyway
A schema guarantees shape, not truth. The total can be a well-formed number that is wrong. Always validate in code (types, ranges, cross-field rules such as "line items sum to total"), and on failure retry with the error message, or route to a person. A minimal version without extra libraries:
import json
from datetime import date
def validate_receipt(raw: str) -> dict:
data = json.loads(raw) # raises on broken JSON
if set(data) != {"merchant", "total", "date"}:
raise ValueError(f"unexpected fields: {sorted(data)}")
if not isinstance(data["merchant"], str) or not data["merchant"]:
raise ValueError("merchant must be a non-empty string")
if not isinstance(data["total"], (int, float)) or data["total"] < 0:
raise ValueError("total must be a non-negative number")
date.fromisoformat(data["date"]) # raises if not YYYY-MM-DD
return data
def extract(call_model, text: str, retries: int = 2) -> dict:
prompt = f"Extract merchant, total, date (YYYY-MM-DD) as JSON.\n\n{text}"
for attempt in range(retries + 1):
raw = call_model(prompt)
try:
return validate_receipt(raw)
except (ValueError, json.JSONDecodeError) as err:
prompt += f"\n\nYour last reply was invalid ({err}). Return only valid JSON."
raise RuntimeError("model never returned a valid receipt")
With a fake model that first returns "total": "about 12.50" and then a correct reply, extract returns {'merchant': 'Cafe Uno', 'total': 12.5, 'date': '2026-11-08'} on the second try. In real code, a schema library (such as Pydantic) does the same checks with less code and can generate the JSON Schema for the API.
Trade-offs and costs
- Constraints can hurt quality. Forcing a format too early can stop the model from "thinking"; a common fix is a
reasoningfield before the answer fields, or letting a reasoning model think first and only constraining the final output. - Schema support varies. Providers support different subsets of JSON Schema (some reject complex unions or recursion). Keep schemas simple and flat where you can.
- Retries cost money and time. With constrained output they become rare; without it, budget for a few percent.
- Enums vs free text. Enums make outputs easy to use and evaluate, but every missing option forces a wrong choice; include "other" or "unknown".
In production
- Log invalid outputs and the validation error; a rising rate often signals a model or prompt change.
- Version schemas alongside prompts; a renamed field breaks consumers silently.
- Streaming: partial JSON can be streamed and parsed incrementally for UIs, but validate the final object.
- Failure modes: truncated JSON when max tokens is too low; valid but empty fields; hallucinated values in required fields the source didn't contain (allow null instead of forcing a value).
When to reach for it
Whenever another program consumes the output: extraction, classification, routing, tool and function calls in agents, filling forms, generating config, and any pipeline between LLM steps.
Example
The expense app switched to schema-constrained output with merchant (string), total (number), currency (enum), date (string, YYYY-MM-DD) and confidence (enum: high, medium, low). Parse failures dropped to zero. Validation still catches about 1% of receipts whose line items don't add up to the total; those go to manual review. Making date nullable stopped the model from inventing dates for receipts without one.
Common mistakes
- Relying on "please reply in JSON" for anything that must not break.
- Trusting a schema-valid reply without checking values.
- Requiring fields the source may not contain, which pushes the model to invent them.
- Huge nested schemas that the provider only partly supports.
- Max tokens too low, cutting the JSON off mid-object.
Say this in the interview
"There are three levels: prompt instructions, JSON mode or tool calling for valid syntax, and schema-constrained decoding, where the decoder masks any token that would break the schema. I'd use schema-constrained output, keep the schema simple, allow null for missing values, and still validate in code with a retry or human review on failure."
Interview questions on this
- How does constrained decoding guarantee valid JSON? It masks tokens that can't continue a valid document according to the grammar, so they can't be sampled.
- Why validate if the output follows the schema? The schema checks shape, not correctness; values can be wrong or invented.
- How can forcing JSON lower answer quality? It removes room for reasoning; add a reasoning field or reason first, then format.
- How do you handle optional information? Make fields nullable and say in the prompt that null is correct when the source doesn't say.
- What's the link between tool calling and structured output? Tool arguments are structured output: the model fills a schema you declared.
Used in
- AI Engineering: Prompt Engineering for Production; Tool Calling and the Agent Loop; Designing Tools an Agent Can Use Well; How an LLM Picks the Next Token.
- AI System Design: Design a Document Processing Pipeline; Design a Customer Support Agent.