A structured-output engine guarantees that the JSON parses and matches the parts of your schema it understands. It guarantees nothing about the values, and some engines skip the parts they don't understand without telling you. So the buying question is what each engine does with a constraint it can't enforce. On a hosted API, use Claude's raw output_config.format, which returns an error on an unsupported keyword, or OpenAI's strict mode, which declined most schemas it couldn't handle in independent testing instead of half-enforcing them. If you self-host on vLLM, SGLang or TensorRT-LLM, switch the grammar backend to llguidance for any schema with patterns, bounds or unions. Whichever engine you pick, you still write the field-level validation, the repair path and the reject path.
The comparison below covers the schema features each engine enforces and its behaviour when it meets one it can't. Licences and docs were checked on each project's live page on October 14, 2026. The open-source engines are free; the hosted modes bill at the model's normal token rates.
| Engine | Where it runs | Licence | When it meets a keyword it can't enforce | Pick it if | Skip it if |
|---|---|---|---|---|---|
| OpenAI Structured Outputs (strict) | OpenAI API | Proprietary API | Supported subset is documented; composition keywords are listed as unsupported | You are on OpenAI and can live inside its subset | You need allOf, if/then/else or optional fields without null unions |
| Claude structured outputs | Anthropic API | Proprietary API | Raw API returns a 400 with details; SDK helpers strip the constraint into the description | You want unsupported keywords rejected with a 400 | You need numeric bounds, minLength or recursion enforced at decode time |
| Gemini structured output | Gemini API | Proprietary API | Docs say "not all JSON Schema features are supported" and don't say whether the rest is ignored or rejected | Your schemas are flat and you validate everything after | You need to know what was enforced without testing it yourself |
| llguidance (and Guidance) | vLLM, SGLang, TensorRT-LLM, llama.cpp | MIT | Fewest under-constrained categories in JSONSchemaBench (1); fails by refusing to compile | You self-host and your schemas are real-world complex | You need XGrammar-only features such as TensorRT-LLM structural tags |
| XGrammar | Default in vLLM, SGLang, TensorRT-LLM, MLC-LLM | Apache-2.0 | Most under-constrained categories in JSONSchemaBench (38): it accepts output the schema forbids | Flat schemas, maximum speed, already the default | Your schema leans on patterns, bounds or unions |
| Outlines | Library over vLLM, transformers, llama.cpp, APIs | Apache-2.0 | 42 compile errors; complex schemas timed out at 40 seconds to 10 minutes | Simple types and regexes in a Python app | You enforce long-tail JSON Schema in production |
| llama.cpp GBNF converter | llama.cpp | Open source | "Unsupported features are skipped silently" | Local or edge inference with simple schemas | You can't run the Python converter to see its warnings |
The loser for production enforcement is Outlines on complex schemas, and XGrammar is the default that most needs watching. Both details come from the same benchmark, covered below.
What Does "100% Schema Adherence" Actually Prove?
It proves the output's structure matches the schema, which removes hallucinated keys and unclosed brackets, and nothing beyond that. OpenAI's launch post reported that gpt-4o-2024-08-06 with Structured Outputs scored 100% on its complex JSON schema-following evals, against under 40% for gpt-4-0613. That is a vendor result on vendor evals, and it measured shape.
Every provider says the same thing in its own docs if you read past the headline. Google's Gemini guide states: "While output is syntactically correct JSON, always validate values in your application", and asks you to handle "schema-compliant but semantically incorrect outputs." Anthropic's docs list the exits from the guarantee: a refusal returns stop_reason: "refusal" with a 200 status, you are billed for it, and the output may not match your schema. A response cut off at max_tokens may not match either. OpenAI's guide makes the same two exceptions and adds a refusal field you must check before parsing.
So a valid invoice object can carry the wrong total, a date in the wrong year, or a customer ID that exists nowhere in your system. Claude's docs add a smaller example that bites in production: Claude "may return a value that differs from your schema only in capitalization" for an enum, so compare enums case-insensitively.
Where Each Engine Sits
Constrained decoding is a technique that masks the model's next-token probabilities so only tokens the grammar allows can be sampled. Every engine here does that. They differ in who runs the mask and how much of JSON Schema they can compile into one.
Provider strict modes run on the vendor's side. You send a schema, the vendor compiles it, and you can't see or swap the engine. OpenAI uses one of the open engines: the llguidance README says OpenAI uses it to power Structured Outputs for JSON Schema.
Runtime engines run inside your inference server. XGrammar (Apache-2.0) describes itself as the default structured-generation backend for vLLM, SGLang, TensorRT-LLM and MLC-LLM. llguidance (MIT, Rust core) ships inside vLLM since v0.8.2, SGLang since v0.4.4, llama.cpp since build b4613, and Chromium. Outlines (Apache-2.0, maintained by .txt) is a Python library that sits over vLLM, Ollama, transformers, llama.cpp and the OpenAI and Gemini APIs. Guidance (MIT) is the higher-level library over llguidance.
On the runtimes themselves: vLLM supports xgrammar and guidance backends with an auto default that picks per request, overridable at startup. SGLang defaults to XGrammar and offers Outlines (no EBNF) and llguidance by flag. TensorRT-LLM offers XGrammar and LLGuidance; only XGrammar supports its structural tags. Our vLLM vs TensorRT-LLM vs SGLang comparison covers the runtime choice itself.
What JSONSchemaBench Found in 10,000 Real Schemas
JSONSchemaBench is the independent evidence to read before choosing. The paper (arXiv 2501.10868) scored Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini on 9,558 real-world schemas from GitHub, Kubernetes, Snowplow, the Washington Post and JSON Schema Store. It measures three things: coverage (which schemas an engine can enforce), efficiency, and the quality of the output.
The coverage numbers in the v3 paper separate the engines on the long tail of the spec:
- On the hardest GitHub set, empirical coverage was 0.41 for Guidance, 0.39 for llama.cpp, 0.28 for XGrammar and 0.03 for Outlines. Guidance had the highest empirical coverage on six of eight datasets.
- On Kubernetes schemas, Guidance reached 0.91 and XGrammar 0.07.
- OpenAI showed low declared coverage on hard sets (0.09 on GitHub Hard, 0.06 on JSON Schema Store) and compliance near 1.00 wherever it accepted the schema. It refuses a schema it can't handle and then follows the ones it accepts, which is the failure mode you want.
- Gemini was omitted on most datasets because it supported under 1% of them. That was the Gemini API the authors tested, and the docs have expanded since, so re-test before relying on that figure in either direction.
The paper's test-suite section is the part that matters for quiet failures. It counted, per engine, the feature categories where the engine threw a compile error, was over-constrained (rejected valid output) or was under-constrained (accepted invalid output). XGrammar was under-constrained in 38 categories, Outlines in 8, llama.cpp in 7 and Guidance in 1. Outlines' low compliance came mostly from timeouts: features such as minItems, maxItems and enum took 40 seconds to 10 minutes to compile.
The authors frame under-constraining as legitimate when an engine does it transparently and leaves the check to the model. In your pipeline it means a JSON object passes the decoder while violating the schema you wrote, and nothing logs it. These results are from the engine versions the paper tested, and all four projects have shipped many releases since, so treat the ranking as a reason to run your own schemas through the current versions.
How Unsupported Schema Features Fail Quietly
A silently dropped constraint is worse than a rejected request, because a 400 shows up in your error rate and a dropped minimum shows up as a negative invoice total three systems downstream. Here is what each engine does with the same workload: a 30-field invoice-extraction schema with a regex on the invoice number, minimum: 0 on amounts, a oneOf between card and wire payment, and a $ref'd address object.
OpenAI strict mode supports pattern, minimum, $defs, $ref and recursion, with limits of 5,000 object properties, 10 levels of nesting and 1,000 enum values. Composition keywords (allOf, not, if/then/else, dependentRequired) are unsupported, oneOf is not on the supported list, and the root can't be anyOf. Every field must be required and additionalProperties must be false. Rewrite the oneOf as a nested anyOf and the invoice schema fits.
Claude returns "a 400 error with details" on the raw API for minimum, maximum, multipleOf, minLength and recursion, and complex regex patterns may get one too. That is loud. Claude's SDK helpers are where the behaviour turns quiet: they remove the unsupported constraint, append it to the field's description, and the Python helper validates the response afterwards. The TypeScript jsonSchemaOutputFormat() helper, per the same page, "doesn't validate the response, so it won't catch violations" such as minimum. On our invoice schema, a TypeScript service could pass negative amounts straight through. There are also hard limits: 20 strict tools, 24 optional parameters and 16 union-typed parameters per request, and a 180-second compile timeout.
Gemini lists minimum and maximum but not pattern, and its examples use $ref and anyOf without listing them as supported. The docs don't say whether an unlisted keyword is ignored or rejected. Test the regex on the invoice number directly before trusting it.
llama.cpp documents the quiet failure plainly: "Unsupported features are skipped silently". Its converter can't mix properties with oneOf, breaks on nested $ref, applies minimum only to integers and needs regexes anchored with ^ and $. Three of our four constraints are at risk. The README recommends running the Python converter to see the warnings.
vLLM adds a version-specific trap: for Qwen3 Coder models, structured outputs "might become disabled if the reasoning content does not get parsed into the reasoning field separately" from v0.11.2. The fix is a config flag, and nothing errors if you miss it.
Does Forcing a Schema Cost Reasoning Accuracy?
Only when you force the schema over the reasoning, and the fix is to let the model reason in free text first and constrain only the final answer. The worry comes from "Let Me Speak Freely?", which reported "a significant decline in LLMs reasoning abilities under format restrictions," worse as the constraint tightened.
The strongest rebuttal comes from .txt, the Outlines maintainer, so weigh its interest. Will Kurt's re-run with matched prompts on Llama-3-8B-Instruct found structured generation scored 0.78 against 0.77 on GSM8K and 0.77 against 0.73 on Last Letter. The post argues the original used different prompts for each mode and a second model to extract answers. JSONSchemaBench reached the same direction independently: on Llama-3.1-8B-Instruct, every constrained engine beat the unconstrained baseline on GSM8K, 83.8% for Guidance against 80.1%.
The design that avoids the tax either way is decoupling. CRANE (ICML 2025) lets the model reason unconstrained, then switches the grammar on for the answer, and reports up to 10 points of accuracy over both constrained and unconstrained baselines on GSM-Symbolic and FOLIO. In production that means one of two things: a reasoning model whose thinking sits outside the schema, or a schema whose first field is a short free-text rationale. Claude's docs warn that a property asking for step-by-step reasoning may trigger a reasoning_extraction refusal, so on Claude use extended thinking and ask the schema for a short explanation at most.
What Constrained Decoding Costs in Throughput
On current engines the token-level cost is small and the compile cost is where you pay. JSONSchemaBench measured time per output token on a llama.cpp backend at 6.37 ms for Guidance against 15.40 ms unconstrained, and about 30 ms for llama.cpp's grammar and Outlines. Guidance gets faster than no constraint because it inserts tokens the grammar already determines instead of sampling them. Outlines took 3.48 to 8.05 seconds to compile a grammar, against near zero for Guidance and llama.cpp. The llguidance README claims about 50 microseconds of CPU per token mask for a 128k tokenizer, a vendor figure.
Batching is where a slow engine hurts everyone. vLLM's January 2025 post explains that mask computation sits on the critical path, so one slow request can delay the whole batch, and reports up to 5x better time per output token under load after moving to XGrammar. The XGrammar paper claims up to 100x over earlier engines.
Speculative decoding is supported but not everywhere at once. NVIDIA and the XGrammar team describe guided decoding working with one-model and two-model speculative decoding in TensorRT-LLM, with the CPU grammar work overlapped with GPU execution. On vLLM, an automated reply in a forum thread on speculation and structured output says all sequences in a batch must have the same proposal length or zero. Test your exact runtime version with speculation on and off and compare both throughput and validity rates.
What Schema Validity Leaves You to Build
Schema validity is a property of the payload's shape. Semantic correctness, meaning the right value in each field, is a property of the model's work, and no engine's marketing separates the two. You need three things that no engine ships:
- Field-level validation in your own code. Re-validate every response against the full original schema, not the subset the engine accepted, then apply business rules: the total equals the sum of line items, the customer ID exists, the date is in range. Pydantic AI and Instructor run validators on the parsed object; Instructor re-asks the model with the error text up to
max_retries. - A repair path with a budget: cap re-asks at one or two, each carrying the validator's error message. Our piece on small models repeating failed tool calls covers why the error format matters for that retry.
- A reject path that routes the record to a queue or a human instead of downstream. Log refusals and
max_tokenstruncations separately from validation failures, because they have different fixes.
How to Decide
The criterion that predicts regret is what happens to a constraint the engine can't enforce. Apart from Outlines' compile times, the measured per-token costs are small enough that speed rarely decides it. Four things change the answer:
- Hosted or self-hosted. On a hosted API you choose by model, not engine, so pick the API whose rejections are loudest for your schemas: OpenAI strict mode or Claude's raw format. If you use Claude's TypeScript helper, add your own validation, since it doesn't run one. Our guide to Claude Opus 5.5's breaking changes to forced tool calls covers a related migration trap.
- Schema complexity. Flat schemas with enums and required fields work on every engine, including the default XGrammar. Once you use patterns, numeric bounds, unions or
$ref, switch vLLM or SGLang to llguidance and re-test. - Who owns the schema. Schemas generated from Kubernetes CRDs or third-party APIs hit the long tail JSONSchemaBench measured. Hand-written schemas can be trimmed to fit an engine.
- Whether you need an LLM at all. If the task is classification into a fixed label set, a smaller decision model may beat a constrained LLM on cost; see our comparison of Jev, Clef and Haiku.
Who should not pick each: skip XGrammar as the only check on any schema with bounds or patterns. Skip Outlines as a production enforcement engine for complex schemas. Skip llama.cpp's converter if you can't run the Python version to read its warnings. Skip Gemini's structured output if you need a documented contract for unlisted keywords, and skip Claude's SDK helpers if you assume they enforce minimum at decode time.
Three Things to Do Before You Wire the Next Agent
This Week: Take your three most complex production schemas and send each one, with a deliberately invalid sample expectation, through your engine. Record which keywords it rejected, which it enforced and which it ignored.
This Month: On vLLM or SGLang, run the same schemas on XGrammar and llguidance and compare validity rates and time per output token. Switch the backend if llguidance enforces more at acceptable speed.
Before Q4 Close: Put a full-schema validator, a two-attempt repair budget and a reject queue in front of every system that consumes model JSON, and add the reject rate to the agent dashboard your agentic stack already reports.
Start with the schema whose minimum would cost you money if it were dropped.
Continue Reading
- vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
- Claude Opus 5.5 Rejects the Forced Tool Calls Opus 5 Accepted
- Jev vs Clef vs Haiku: Decision Models Cut Cost 25x, Not Errors
- Wiping the Transcript Was the Worst Retry. Paraphrase.
- NeMo Guardrails vs Guardrails AI vs Lakera: Buy the Detector
- The 2026 Agentic AI Stack: 8 Layers, 3 You Can Skip