# How to serve schema-constrained output with vLLM > Pin a decoding backend, send the schema in the field this vLLM release reads, and run a harness that validates every reply and refuses a silent fallback. Source: https://voxgig.com/howto/serve-schema-constrained-output-with-vllm - Audience: platform-team - Level: advanced - Languages: python - Verified: 2026-10-21 - Published: 2026-10-19 ## Short answer Pin `--structured-outputs-config.backend` to `xgrammar` or `guidance` on `vllm serve`, so a schema the engine cannot take fails the request with a 400 instead of moving to another backend under `auto`. Send the schema as `structured_outputs` or `response_format`; the removed `guided_json` family is ignored with a warning. Preflight every schema with llguidance offline, and validate every reply against the schema before you read the server's throughput. --- ## You will need A GPU host already running vLLM, and Python 3.11 or later for the harness. Verified 2026-10-21 against Python 3.13.16, `openai` 3.35.1, `jsonschema` 4.26.0, and `llguidance` 1.9.1. vLLM 0.32.0 and SGLang 0.5.22 were read in their documentation and source, not run, because there is no GPU here. The server in every output block is a stand-in on 127.0.0.1. It answers vLLM's request shapes the way the source does and compiles schemas with the same llguidance engine, so the counts are real and the speeds are not. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [SGLang](https://docs.sglang.io/docs/advanced_features/structured_outputs) | A second serving stack is on the table, and you want XGrammar by default with llguidance one flag away | Another server to operate and tune, its own request fields for regular expressions and EBNF, and no `auto` selection to inherit | You are staying on vLLM, and the question is which backend rather than which server | | [The vLLM `guidance` backend](https://docs.vllm.ai/en/latest/features/structured_outputs/) | Schemas that change per request, or keywords vLLM refuses for XGrammar, such as `multipleOf` | A mask computed per token on the CPU, and a model free to add keys unless the schema or the `disable_additional_properties` flag forbids them | A few schemas serve every request, where a compiled grammar pays for itself | | [The vLLM `xgrammar` backend](https://docs.vllm.ai/en/latest/features/structured_outputs/) | A few schemas reused across many requests, compiled once and served from the cache | A compile step per new grammar, a cache to size, and a list of JSON Schema keywords it refuses outright | Your schemas use `multipleOf`, `uniqueItems`, or a `pattern` with a length bound | XGrammar compiles a grammar ahead of time and caches it, so a repeated schema is cheap and a stream of new ones pays the compile each time. llguidance skips the compile and charges per token instead, about fifty microseconds a mask by its own measure, which is why it suits schemas that differ per request. SGLang offers the same two engines behind one flag, at the price of a second server with its own request fields. ## Compile every schema before it reaches a GPU llguidance is the engine behind vLLM's `guidance` backend, and its validator needs no tokenizer and no model. Run every schema through it in CI, before a request can reach the server. ```py title="preflight.py" def check(schema): """'' when the engine compiles the schema, otherwise its first error message.""" grammar = llguidance.LLMatcher.grammar_from_json_schema(schema) is_error, messages = llguidance.LLMatcher.validate_grammar_with_warnings(grammar) return messages[0] if is_error else "" ``` The four schemas here are an order, an invoice with `multipleOf: 0.01` on its total, a contact with a `pattern` and a `maxLength` on one field, and a tag list with `uniqueItems`. ```bash PYTHONPATH=vendor python3 preflight.py schemas ``` ```text output schema guidance engine (llguidance 1.9.1) contact ok invoice ok order ok tags rejected: Unimplemented keys: ["uniqueItems"] 1 of 4 schemas would be refused with the backend pinned to guidance ``` The engine takes three of the four and names the keyword it refuses. XGrammar draws the line elsewhere. vLLM's own pre-check for that backend, in [`backend_xgrammar.py`](https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/v1/structured_output/backend_xgrammar.py), refuses `multipleOf` on a number, `uniqueItems` and `contains` on an array, and a `pattern` or `format` combined with a length bound. It also refuses several `propertyNames` and `patternProperties` combinations, and every refusal carries one message, `The provided JSON schema contains features not supported by xgrammar.` in the 400 body. So the invoice and the contact pass llguidance and fail XGrammar, and the tag list fails both.[^1] Under `auto`, vLLM tries XGrammar first and, when that check fails, hands the request to guidance, or to outlines when guidance's own check refuses it, as [`sampling_params.py`](https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/sampling_params.py) spells out. With a backend named, the same code comments `xgrammar with no fallback` and the request fails. ## Pin the backend and send the schema where this release reads it Name the backend on the server. The [`vllm serve`](https://docs.vllm.ai/en/latest/cli/serve/) option takes either dotted keys or one JSON string, and its fields are documented on the [`StructuredOutputsConfig`](https://docs.vllm.ai/en/latest/api/vllm/config/structured_outputs/) page. This command runs on the GPU host and was not run here: ```bash vllm serve Qwen/Qwen3-8B --structured-outputs-config '{"backend": "xgrammar", "disable_any_whitespace": true}' ``` On the client, the schema travels in one of two fields. `structured_outputs` is vLLM's own, sent through the [`openai`](https://github.com/openai/openai-python) client's `extra_body` because the client has no parameter for it; `response_format` with `json_schema` is the OpenAI shape, which vLLM maps onto the same constraint. ```py title="client.py" def request_kwargs(schema, shape): """The keyword arguments that put the schema on the wire in the given shape.""" if shape == "structured_outputs": return {"extra_body": {"structured_outputs": {"json": schema}}} if shape == "response_format": name = schema.get("title", "reply") return {"response_format": {"type": "json_schema", "json_schema": {"name": name, "schema": schema}}} if shape == "guided_json": return {"extra_body": {"guided_json": schema}} raise ValueError(f"unknown shape {shape!r}") ``` The third shape is the trap. The `guided_*` fields were removed, and the [request model](https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/entrypoints/serve/engine/protocol.py) keeps a list of their names so it can log, once, that they were ignored. The demo sends the order schema in all three shapes to the stand-in and validates each reply. ```bash PYTHONPATH=vendor python3 shapes.py ``` ```text output shape field on the wire server reply valid structured_outputs structured_outputs constrained True response_format response_format constrained True guided_json guided_json ignored False server log: Request contains the removed guided-decoding field(s) ['guided_json'], which are ignored; output will NOT be constrained. ``` The first column is what the `openai` client put on the wire. The last is what a request with the old field gets back: a 200 with prose in it, and a warning in a log the client never sees. ## Count a reply only after it validates The harness sends each schema several times, validates every reply with [`jsonschema`](https://python-jsonschema.readthedocs.io/en/stable/) before counting it, and counts a 400 as a rejected schema rather than a skipped request. ```py title="measure.py" for _ in range(requests): try: content, usage = complete(client, model, schema, prompt, shape) except BadRequestError as err: counts["rejected"] += 1 counts["reason"] = str(err.body.get("message", err.message) if isinstance(err.body, dict) else err.message) continue counts["tokens"] += usage.completion_tokens if usage else 0 try: value = json.loads(content) validator.validate(value) counts["valid"] += 1 except (json.JSONDecodeError, jsonschema.ValidationError): counts["invalid"] += 1 ``` Against the stand-in, which behaves as a server pinned to `guidance`, the table proves the accounting. Against the GPU host, pass `--base-url` and `--model`, and the last line becomes completion tokens per second. ```bash PYTHONPATH=vendor python3 measure.py --standin ``` ```text output shape: structured_outputs, requests per schema: 5 schema valid invalid rejected tokens time contact 5 0 0 90 duration_ms: 56.2 invoice 5 0 0 80 duration_ms: 15.9 order 5 0 0 140 duration_ms: 13.7 tags 0 0 5 0 duration_ms: 14.4 400: Grammar error: Unimplemented keys: ["uniqueItems"] tokens per second: not measured, the stand-in has no model behind it ``` The tag schema shows what a pinned backend does with an unsupported schema: every one of its five requests is refused, with the engine's reason under the row. That is the loud failure you want in a test run. Measure throughput with the schemas your agents send, not with a toy. The per-token cost of the guidance engine and the compile cost of XGrammar both scale with the grammar, and the XGrammar [cache](https://xgrammar.mlc.ai/docs/latest/start/workflow_of_xgrammar.html) only helps when the same grammar comes back.[^2] ## Check it worked Thirteen checks run the harness's own logic against the stand-in. Each raises on failure rather than using `assert`, which `python -O` strips. ```bash PYTHONPATH=vendor python3 test_harness.py ``` ```text output ok preflight accepts order, invoice, and contact ok preflight rejects tags and names the keyword ok structured_outputs travels in extra_body ok response_format carries the schema under json_schema ok guided_json is a removed field, not a parameter ok the server received the structured_outputs field ok the server received the response_format field ok the server received the guided_json field ok a constrained reply that matches its schema counts as valid ok an unconstrained reply counts as invalid, never valid ok the server warned that the removed field was ignored ok a refused schema counts as rejected with the engine's reason ok a refused schema is never counted valid 13 checks passed ``` The two that matter most are `an unconstrained reply counts as invalid, never valid` and `a refused schema is never counted valid`, because a harness that gets either wrong reports a healthy endpoint that is not constraining anything. ## When it goes wrong A schema that works in staging returns 400 in production. Staging ran `auto`, so the request moved to another backend when the first refused it, and production pins one. Pin the same backend in both, and run the preflight for that backend in CI so the refusal happens before a deploy. Replies stop being JSON after a vLLM upgrade. The client still sends a `guided_*` field, the server ignores it, and the warning appears once in the server log. Switch to `structured_outputs` or `response_format`, and keep the harness in CI so the `invalid` column catches the next one. Throughput falls when a new agent joins. Its schemas differ on every request, so XGrammar compiles a grammar per request and its cache, which evicts by least recent use, stops hitting. Move that workload to `guidance`, which compiles nothing up front, or give the agent a fixed set of schemas. ## When not to do this Do not run `auto` where you need to know which engine decoded a reply. The selection happens per request, is logged nowhere the client can see, and changes with each release, as the config's own docstring says. A pinned backend fails a schema out loud, which is the behavior a test can hold on to. Do not take a 200 as a constrained reply. The removed `guided_*` fields produce exactly that, a 200 with a warning in a log you are not reading. Validate on the client, every reply, even on the day the server is known to be right. Do not treat the grammar as validation of meaning. A grammar guarantees shape, so an invoice total of `0.00` and a contact at `nobody@example.com` are both valid output. Validators for the values that matter still run after the parse. Do not benchmark a backend with one small schema. The engine's cost is a function of the grammar and of how often it changes, and a toy schema measures neither. Run the harness with the schemas your agents send and with the traffic shape they produce.[^3] ## Related how-tos - [Validate JSON model output before you use it](/howto/validate-json-model-output-in-typescript) - [Compare agent task success rates across model providers](/howto/measure-task-success-across-model-providers) ## Last verified Verified 2026-10-21 against Python 3.13.16, `openai` 3.35.1, `jsonschema` 4.26.0, and `llguidance` 1.9.1. Every output block is what the command preceding it printed. vLLM 0.32.0 and SGLang 0.5.22 were not run. Their backends, flags, request fields, fallback order, and warning text were read from their documentation and source at those versions. The stand-in reproduces that behavior for the three request shapes with the llguidance engine. XGrammar's refusals are quoted from vLLM's pre-check and were not exercised, because the package needs a GPU runtime this machine lacks. [^1]: The two engines disagree about `multipleOf` for a reason each would call principled. XGrammar's converter logs a warning and ignores the keyword, and vLLM's [pre-check](https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/v1/structured_output/backend_xgrammar.py) quotes that warning in its source and refuses the schema rather than let a constraint vanish. llguidance's [converter](https://github.com/guidance-ai/llguidance/blob/main/docs/json_schema.md) lists the keyword as supported for both integers and numbers. An invoice total with two decimal places is therefore a 400 from one backend and a constraint on the other. Under `auto` it is whichever of the two the request happened to reach. [^2]: The figure of about fifty microseconds a mask is llguidance's own, from its [README](https://github.com/guidance-ai/llguidance), measured for a tokenizer with 128k tokens. It sits beside the claim that there is no significant startup cost. The same README says XGrammar's pre-computation often runs into seconds and sometimes minutes, which is a competitor's measurement and reads like one. XGrammar's documentation describes the compile step and its LRU cache and gives no timings, so the only numbers on the table are from the side that computes nothing in advance. [^3]: The `guided_json` family was removed in v0.12.0, according to the [structured outputs guide](https://docs.vllm.ai/en/latest/features/structured_outputs/), which still lists the six names so a reader can recognize them in old code. The request model keeps the same six in a set called `_REMOVED_GUIDED_FIELDS`, to warn when one arrives. A feature that has been removed and still has a list of its own names in the code is a feature the authors expect to keep receiving.