High Function calling Question 175 of 223

How does constrained decoding differ from JSON mode?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

PICTURE THIS: A SENTENCE BECOMES TOKENS

The model does not read letters like humans. It reads these pieces, then predicts the next one.

Simple meaning

JSON mode asks the model to emit JSON but may still be invalid or off-schema.

WHY — Tokens instead of words?

LLMs use tokens (not full words) because it helps them:

This is a process

question about Function calling.

Panels listen for order,

trade-offs, and what you would actually do on a GenAI / LLM project - not buzzwords.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

STEPS — What happens with tokens?

Before the model can read a sentence, it goes through these steps:

1
JSON mode asks the
model to emit JSON but may still be invalid or off-schema.
2
Constrained decoding, including grammar
or FSMs, only allows tokens that keep the output in a formal language.
3
Embeddings
Constraints raise validity
4
they do not guarantee
truthful field values.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

EXAMPLE — See it in action

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“Constrained decoding, including grammar or FSMs, only allows tokens that keep th”

Tokenized output

ConstraineddecodingincludinggrammarorFSMs

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

Key takeaway

JSON mode asks the model to emit JSON but may still be invalid or off-schema. Constrained decoding, including grammar or FSMs, only allows tokens that keep the output in a formal language.