The Grammar Is the Guarantee

Prompting a model to "respond only with valid JSON" is a request. The model can refuse it — not intentionally, but because token-by-token sampling has no memory of your instruction once a constraint is violated. You get near-valid JSON, almost-valid JSON, JSON with a trailing comma. You get a retry budget.

Constrained decoding is the alternative.

How It Works

At each decode step, the inference engine computes a token mask — a bitmask over the entire vocabulary. Any token that would make the partial output invalid against the grammar is set to logit -∞ before sampling. The model never sees those tokens as candidates. It cannot select them. The output is not shaped by persuasion; it is shaped by elimination.

The constraint is compiled from a formal specification:

• JSON Schema → compiles to a finite state machine (FSM) that tracks which keys have been emitted, which types are expected, which brackets are open.
• Regex → compiles to an FSM over the character sequence.
• EBNF / GBNF grammar → compiles to a pushdown automaton, allowing recursive and context-sensitive structure (valid SQL, valid Python, custom DSLs).

The primitive in every case is identical: logits[invalid_tokens] = -∞. The sophistication lives in the compiler, not the sampler.

The Tools

llama.cpp introduced GBNF (GGML BNF), a format for defining context-free grammars that constrain local inference. Any schema convertible to EBNF can be enforced. The overhead is real — grammar sampling checks candidates against the constraint state on every step — but benchmarks on quantized models put it at roughly 8–15% for well-written grammars. Pathological patterns (unbounded optional repetitions like x? x? x?) can degrade severely; the fix is x{0,N}.

Outlines (Python, model-agnostic) provides a clean interface over the same mechanism. Pass a JSON Schema, a regex, or an EBNF grammar; get a guided sampler back. It works across Transformers, llama.cpp backends, and vLLM.

XGrammar (used in vLLM v1) makes a key optimization: it classifies vocabulary tokens into context-independent tokens (those whose validity does not depend on the current parser state) and context-dependent tokens. Context-independent tokens — typically more than 99% of the vocabulary — can be precomputed once at grammar compilation time. Per-step mask generation then only evaluates the small context-dependent set. The result is up to 80x throughput improvement over naive FSM-per-token evaluation.

Why This Matters More for Small Models

Large frontier models follow instructions well enough that prompt-based JSON elicitation has an acceptable success rate. Models below 13B parameters are weaker instruction-followers. For self-hosted inference on a 7B or 8B model, constrained decoding is not a nice-to-have — it is what makes the output usable in a pipeline without a retry loop.

The constraint does not improve the model's reasoning. It does not help the model choose the right values. It only guarantees the format. That is enough to eliminate an entire class of production failures.

The Structural Insight

A prompt is a prior. A grammar is a hard constraint. They operate at different layers of the generation process, and conflating them leads to architectures that are fragile by design.

If your pipeline retries on parse error, you have a prompt where you need a grammar.

References

• Dong, Y., Ruan, C. F., Cai, Y., Lai, R., Xu, Z., Zhao, Y., & Chen, T. (2024). XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. MLSys 2025. https://proceedings.mlsys.org/paper_files/paper/2025/file/5c20ca4b0b20b0bd2f1d839dc605e70f-Paper-Conference.pdf
• MLC AI Team. (2024). Achieving Efficient, Flexible, and Portable Structured Generation with XGrammar. MLC Blog. https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar
• ggml-org. (2024). GBNF Grammar Format. llama.cpp documentation. https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md
• Bansal, J. (2026). Constrained Decoding: Grammars, Regex, and FSMs. jatinbansal.com. https://jatinbansal.com/ai-engineering/constrained-decoding/
• BentoML. (2025). Structured Decoding in vLLM: A Gentle Introduction. https://www.bentoml.com/blog/structured-decoding-in-vllm-a-gentle-introduction
• mvpfactory.io. (2026). Structured output grammars for on-device LLMs on Android. https://mvpfactory.io/blog/structured-output-grammars-for-on-device-llms-constraining-llama-cpp-token/

devinfo.dev. (2026). "The Grammar Is the Guarantee." devinfo.dev:2026.0060. https://devinfo.dev/d/2026.0060