JSON Schema for LLM Structured Outputs

Structured outputs constrain an LLM’s response to a JSON Schema you supply, so the model returns machine-readable JSON that matches your fields, types, and enums instead of freeform prose you have to parse by hand.

Anyone who has shipped an LLM feature knows the alternative: a regex to strip stray markdown fences, a try/catch around JSON.parse, and a retry loop that fires more often than you would like. It holds up fine until the morning it quietly doesn’t.

The provider enforces the schema during generation, which turns “hope the model returns valid JSON” into a contract. This works today across OpenAI, Google Gemini, and Anthropic Claude. All three accept JSON Schema, so the same schema is portable and you change only the request wiring, not the contract. This article covers what structured outputs are, how JSON Schema drives them, how enforcement works under the hood, how to wire it up per provider, and the failure modes worth guarding against.

From freeform text to schema-enforced JSON

Left to their own devices, LLMs emit freeform text that breaks parsers: they add prose around the JSON, omit quotes, or invent fields. Structured outputs solve this by constraining the response to a JSON Schema, so the output is machine-readable and reliably parseable.

This is the upgrade over the older JSON mode. OpenAI introduced JSON mode in 2023 as a way to force valid JSON, but it only promises that the output will parse, not that it will follow any schema you define. Strict structured outputs close that gap by enforcing the schema itself. On OpenAI’s own evaluation of complex schema-following, the gpt-4o-2024-08-06 model with Structured Outputs scored 100%, compared with under 40% for the older gpt-4-0613. That is a model-specific benchmark, not a universal guarantee, but it illustrates the shift from “usually parses” to “matches the schema.”

How does JSON Schema fit in?

A JSON Schema is a declarative contract for your data: it declares types, required fields, enums, and value constraints. You pass it with the request, and the provider constrains generation to match it. A compact schema for a contact extraction looks like this:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "email": { "type": "string" },
    "plan_interest": {
      "type": "string",
      "enum": ["starter", "pro", "enterprise"]
    }
  },
  "required": ["name", "email", "plan_interest"],
  "additionalProperties": false
}

Few people hand-write these in production. The common workflow is to define the shape in Pydantic (Python) or Zod (TypeScript) and let the SDK emit JSON Schema. OpenAI’s SDKs support this directly: hand them a Pydantic or Zod object and they generate the matching JSON schema, turn the response back into your typed object, and surface refusals for you. Gemini added the same convenience, extending JSON Schema support to every actively supported Gemini model so Pydantic and Zod schemas work without a conversion step.

If you would rather not hand-assemble the schema or the per-provider wrapper, OpenReplay’s JSON Schema builder does both in the browser. You add fields with types, descriptions, enums and nesting, or paste a sample JSON to infer a starting point, then copy the result from the export panel, which offers JSON Schema, OpenAI response_format, OpenAI function tool, Anthropic, Gemini, Zod and Pydantic. Nothing you type leaves the page.

How does schema enforcement work?

Structured outputs work because the provider constrains decoding: at each generation step, the model can only produce tokens that keep the output valid against your schema. The schema is compiled into a grammar (or a finite-state machine), and tokens that would violate it are masked out before sampling. Anthropic’s own docs describe the same mechanism: your schema is compiled into a grammar, and constrained sampling keeps generation inside it, so the model has no way to emit a token that would break the schema.

The same idea generalizes beyond JSON. For local and self-hosted models, llama.cpp uses GBNF grammar files and Outlines applies regex- and grammar-based constraints, both enforcing arbitrary formats (SQL, custom DSLs, or JSON) by the same token-masking principle.

Structured outputs across providers

All three major providers speak JSON Schema, so the pattern ports. What differs is the request wiring and the failure signals.

OpenAI. Set strict: true and pass the schema. OpenAI’s guide recommends starting new projects on its current models and notes that the Responses API moved the parameter: use response_format: { type: "json_schema", strict: true } on Chat Completions, or text: { format: { type: "json_schema", strict: true } } on the Responses API.

Gemini. Supply the schema through generationConfig:

response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents="Extract the contact from this email...",
    config={
        "response_format": {
            "text": {
                "mime_type": "application/json",
                "schema": person_schema,
            }
        },
    },
)

Claude. Anthropic ships native structured outputs, no longer a tool-call workaround. There are two complementary features: JSON outputs via output_config.format for the response body, and strict tool use via strict: true for tool inputs, usable independently or together. Strict tool use guarantees that a call’s arguments match its input_schema, because that schema is compiled into a grammar that constrains sampling, the same family of techniques OpenAI and Gemini use. Note the API surface changed at GA: the output_format parameter moved to output_config.format, and beta headers are no longer required.

Gotchas and best practices

Strict mode is not a guarantee of parseable output, and schema support is not universal. Guard against these.

Keep schemas flat. Deeply nested or recursive structures are the most common cause of “schema too complex” errors and degraded reasoning. Claude surfaces this directly, returning a 400 when the compiled grammar grows too large, and Gemini’s docs warn that very large or deeply nested schemas may be rejected. Long property names, large arrays, enums with many values, and objects full of optional properties all add to the cost. Break large extractions into smaller, flatter schemas.

Handle refusals and truncation. Even with strict mode on, a refusal or a length-truncated response returns a success status but is not schema-valid JSON. OpenAI added a dedicated signal for this: a refusal field on the response tells you the model declined rather than returning something that matches your schema. There is no universal output-token limit. Caps vary by model, and any response that hits the cap mid-object yields invalid JSON, so size your max_tokens for the worst case and check your model’s documented ceiling.

Verify the supported subset. Every provider supports only a slice of JSON Schema in strict mode. OpenAI’s guide is explicit that a good deal of the spec is covered but some parts are left out, either for performance or for technical reasons. Keywords like minimum, pattern, or default values may be dropped or rejected, so check the provider’s supported-subset docs rather than assuming the full spec.

Validate anyway. Because refusals and truncation produce valid-status-but-invalid-JSON responses, parse and validate the object against your schema before trusting it, even with strict: true.

Reason first, then emit. Constraining output can reduce reasoning quality on some tasks. One practitioner guide to Claude’s structured outputs treats this as a real tradeoff against extended thinking: if a task gains more from the model’s reasoning than from guaranteed schema compliance, leave the thinking unconstrained. A practical middle path is to let the model reason in a thinking phase, then constrain only the final JSON.

Structured outputs turn LLM responses into something you can treat like a typed API, and the pattern moves cleanly across OpenAI, Gemini, and Claude because they all accept JSON Schema. Start by defining your shape in Pydantic or Zod, enable strict mode on your provider, keep the schema flat, and wrap the parse in validation that handles refusals and truncation, then wire the same schema into whichever provider you deploy against.

FAQs

What is the difference between JSON mode and strict structured outputs?

JSON mode only guarantees the model returns syntactically valid JSON that parses without errors, but it does not guarantee the output matches any particular schema. Strict structured outputs enforce your specific JSON Schema during generation, so the returned object has the fields, types, and enums you defined. The distinction is 'it parses' versus 'it has the fields you need.' OpenAI introduced JSON mode in 2023 and later added schema-enforced structured outputs to close that gap.

Does enabling strict mode guarantee you always get valid, parseable JSON?

No. Strict mode constrains token generation to your schema during normal completion, but a safety refusal or a length-truncated response still returns a success status while producing output that is not schema-valid JSON. OpenAI exposes a dedicated refusal field and a finish_reason of length; Claude signals these with a stop_reason of refusal or max_tokens. Because these responses return a 200 and get billed, you should parse and validate the object against your schema before trusting it.

Can I reuse the same JSON Schema across OpenAI, Gemini, and Claude?

Largely yes. OpenAI, Google Gemini, and Anthropic Claude all accept JSON Schema, so the same schema is portable and you change only the request wiring, not the contract. What differs is where the schema goes: OpenAI uses response_format or text.format with strict true, Gemini nests the schema under responseFormat.text in generationConfig, and Claude uses output_config.format or strict tool use. Each provider supports only a subset of JSON Schema, so verify unsupported keywords against each provider's supported-subset docs before assuming full portability.

Why does my schema get rejected as too complex, and how do I fix it?

Complexity limits come from deeply nested or recursive structures, long property names, large array limits, enums with many values, or objects with many optional properties. Claude returns a 400 when the compiled grammar grows too large, and Gemini may reject very large or deeply nested schemas. The fix is to keep schemas flat and break large extractions into smaller, flatter schemas. Deeply nested structures are also a common cause of degraded reasoning, so flattening improves both acceptance and output quality.