AskpertDoes JSON output make LLM reasoning worse?
A widely cited paper says format restrictions wreck reasoning. A direct rebuttal says the experiment was broken. Both sides agree on one engineering rule, and it is the one worth shipping.
On this page
Terms, definedthe jargon, decoded
- JSON mode
- A provider setting that pushes a model to return syntactically valid JSON. It does not necessarily enforce your schema.
- Constrained decoding
- Restricting which tokens are legal at each step so output must match a grammar or schema. A different mechanism from JSON mode, and often confused with it.
- Format-restricting instruction
- Simply telling the model in the prompt to answer in a given format, with no enforcement at decode time.
- Parse validity
- Whether output can be read by your parser at all, which is a separate measurement from whether the answer is correct.
- Last Letter Concatenation
- A benchmark task that asks the model to take the last letter of each name in a list and join them, used to test multi-step symbolic work.
Structured output changes more than the shape of a response. JSON mode, function calling and constrained decoding all restrict which tokens are available during generation. The question that decides whether that costs you anything is whether the model has to reason and serialize under the same constraint.
The published evidence is contested, and the disagreement is more useful than either side alone. Tam et al. reported substantial losses under format restrictions on some tasks. dottxt's direct response identified experimental defects and reported a matched re-run in which structured generation came out ahead. Neither supports treating JSON mode as universally safe or universally damaging.
What did the format-restriction study find?
Tam et al. tested five models across three conditions: JSON mode, format-restricting instructions, and natural-language to format conversion. It measured task performance, not just whether the response parsed.
On GSM8K, GPT-3.5-Turbo scored 75.99% with natural-language answers, 74.70% with format-restricting instructions, and 49.25% in JSON mode. Claude-3-Haiku scored 86.51% as text against 23.44% under a strict JSON schema. On Last Letter Concatenation, GPT-3.5-Turbo scored 56.7% with natural language and 25.2% with format-restricting instructions.
Then it reverses. On DDXPlus classification, GPT-3.5-Turbo scored 44.1% with natural language and 55.5% in JSON mode.
That reversal is the most informative number in the paper, because a single blanket JSON penalty cannot explain a result where JSON mode improves the task. Tam et al.'s own recommendation follows from it: answer in prose, then convert the answer into the required format.
Why did dottxt say the experiment was broken?
The rebuttal's central charge is that the comparison never isolated formatting. The natural-language and JSON conditions used different prompts, and the JSON prompt omitted the schema. Asking for JSON without showing the required structure is not the same request as asking for JSON and supplying it.
There is also a parsing problem. Tam et al. used Claude-3-Haiku as a "perfect text parser" for one task instead of a deterministic parser such as a regular expression, which folds parser behavior into a generation result. And the one-shot example used two names while test items contained four, so the demonstration did not match the evaluation.
The rebuttal's last point is conceptual and it is the one most likely to bite you in production: JSON mode is not constrained generation. JSON mode may bias a model toward valid JSON without enforcing every schema constraint, while constrained decoding changes the set of legal next tokens. Treating them as one condition makes any causal claim imprecise.
What happened when the prompts were matched?
On Last Letter Concatenation with matched prompts, dottxt reported 0.68 for natural-language structured output, 0.73 for unstructured JSON, and 0.77 for structured JSON. Structured generation beat both comparisons.
This does not prove constrained decoding always improves reasoning. It establishes something narrower and more useful: a gap attributed to JSON may actually be a prompt-quality gap, a parser-design gap, or a mismatch between JSON mode and schema-constrained generation. Any claim about structured output is only as good as its exact prompt, schema, decoder, parser, demonstrations and scoring procedure.
The practical warning is that both "JSON is fine" and "JSON is fatal" are currently being repeated by people who have not read either setup.
Is JSON mode safe for every task?
JSON output is safest when the model maps input to short, local fields without exposing a long intermediate derivation. Extraction, classification labels, routing decisions and short factual fields all fit: the contract narrows the answer space without asking the model to carry a chain of intermediate state through the serialized response.
Two-pass generation is the safer default for multi-step reasoning, mathematics, planning, and anywhere an open-text derivation improves accuracy. Where an open derivation genuinely improves accuracy is itself measured: a meta-analysis found chain-of-thought's gains concentrated in math and symbolic tasks. Ask for the working in ordinary text, then hand that result to a formatter and require validation. The second pass should serialize the answer, not reconsider it.
| Task shape | One constrained pass | Two passes | Reason |
|---|---|---|---|
| Entity extraction | Usually safe | Usually unnecessary | Fields are local and directly grounded in the input |
| Classification | Usually safe | Useful when the label depends on analysis | The label set is small, but the decision may not be |
| Short factual fields | Usually safe | Useful for ambiguous inputs | Serialization adds little reasoning load |
| Multi-step mathematics | Risky | Preferred | Intermediate derivation can affect the result |
| Tool or workflow planning | Risky | Preferred | The plan requires several dependent decisions |
| Long-form analysis | Risky | Preferred | The response needs open-ended structure before serialization |
This is an engineering decision rule, not a benchmark finding. The published studies do not establish a threshold at which every model should switch from one pass to two.
How should you test whether a schema costs accuracy?
Test your production task with equivalent prompts and an explicit schema in every structured condition. Hold the model, temperature, examples, input distribution and scoring code constant, then compare natural-language output, instructed JSON, JSON mode, and real constrained decoding if your serving stack supports it.
Keep three measurements apart: task correctness, parse validity, and schema validity. A response can be semantically correct and fail a parser. A response can be valid JSON carrying the wrong classification. Collapsing them into one score hides the failure mode you actually have to operate against.
Use deterministic parsing wherever the format permits, and if a parser is genuinely required, test it separately against malformed, partial and unexpected outputs. This is the same boundary that makes tool selection accuracy a different problem from output validity, since a valid object does not establish that the model chose the right action, and the reason reliability depends on the property being measured rather than on one aggregate score.
What advice survives both papers?
Treat structured output as an interface constraint rather than a theory of reasoning. Use a schema directly for extraction, classification, routing, or short factual mapping. Use two passes when the task needs several dependent inferences, arithmetic, planning, or analysis whose intermediate structure changes the answer.
And do not infer a reasoning penalty from an unreviewed benchmark. Read the prompts and schemas, check whether the parser was deterministic, distinguish JSON mode from constrained decoding, then reproduce the task on your own inputs and score semantic correctness separately from serialization.
Constrain the response when serialization is the task. Postpone it when reasoning is the task. That rule holds even if the next benchmark disagrees about the average effect of JSON.
Does JSON mode reduce LLM accuracy?
It can on some tasks, but the effect is not universal. Tam et al. reported large losses on selected reasoning benchmarks, while a dottxt re-run with matched prompts reported higher scores for structured generation on Last Letter Concatenation. The result depends on the task, prompt, schema, decoder, parser, model, and scoring method.
Should an LLM reason in JSON?
Use JSON in one pass for direct extraction, classification, routing, or short factual fields. Use open text for multi-step reasoning, mathematics, planning, or analysis, then serialize the finished result in a second pass. The separation reduces how many responsibilities land on a single generation.
Is JSON mode the same as constrained decoding?
No. JSON mode encourages valid JSON or applies provider-specific restrictions. Constrained decoding limits the legal next tokens according to a grammar or schema. A benchmark has to say which one it tested before its result can be carried to another serving stack.
Does structured output improve reasoning?
It has no fixed effect. The dottxt matched re-run reported Last Letter scores of 0.68 for natural-language structured output, 0.73 for unstructured JSON, and 0.77 for structured JSON. That shows careful prompts and constraints can beat a comparison condition, not that every schema improves every task.
How can engineers test JSON output fairly?
Keep the model, inputs, demonstrations, sampling settings, schema, and evaluator consistent across conditions. Score semantic correctness, parse validity, and schema validity separately. Use deterministic parsing where possible, and test JSON mode independently from true constrained decoding. Publish the exact prompt and schema so others can reproduce the comparison.