Use Ctrl+P (or Cmd+P) to save as PDF. Back to paper
Prompting a model to "respond only with valid JSON" is a request. The model can refuse it — not intentionally, but because token-by-token sampling has no memory of your instruction once a constraint is violated. You get near-valid JSON, almost-valid JSON, JSON with a trailing comma. You get a retry budget.
Constrained decoding is the alternative.
At each decode step, the inference engine computes a token mask — a bitmask over the entire vocabulary. Any token that would make the partial output invalid against the grammar is set to logit -∞ before sampling. The model never sees those tokens as candidates. It cannot select them. The output is not shaped by persuasion; it is shaped by elimination.
The constraint is compiled from a formal specification:
The primitive in every case is identical: logits[invalid_tokens] = -∞. The sophistication lives in the compiler, not the sampler.
llama.cpp introduced GBNF (GGML BNF), a format for defining context-free grammars that constrain local inference. Any schema convertible to EBNF can be enforced. The overhead is real — grammar sampling checks candidates against the constraint state on every step — but benchmarks on quantized models put it at roughly 8–15% for well-written grammars. Pathological patterns (unbounded optional repetitions like x? x? x?) can degrade severely; the fix is x{0,N}.
Outlines (Python, model-agnostic) provides a clean interface over the same mechanism. Pass a JSON Schema, a regex, or an EBNF grammar; get a guided sampler back. It works across Transformers, llama.cpp backends, and vLLM.
XGrammar (used in vLLM v1) makes a key optimization: it classifies vocabulary tokens into context-independent tokens (those whose validity does not depend on the current parser state) and context-dependent tokens. Context-independent tokens — typically more than 99% of the vocabulary — can be precomputed once at grammar compilation time. Per-step mask generation then only evaluates the small context-dependent set. The result is up to 80x throughput improvement over naive FSM-per-token evaluation.
Large frontier models follow instructions well enough that prompt-based JSON elicitation has an acceptable success rate. Models below 13B parameters are weaker instruction-followers. For self-hosted inference on a 7B or 8B model, constrained decoding is not a nice-to-have — it is what makes the output usable in a pipeline without a retry loop.
The constraint does not improve the model's reasoning. It does not help the model choose the right values. It only guarantees the format. That is enough to eliminate an entire class of production failures.
A prompt is a prior. A grammar is a hard constraint. They operate at different layers of the generation process, and conflating them leads to architectures that are fragile by design.
If your pipeline retries on parse error, you have a prompt where you need a grammar.