AskpertWhen chain-of-thought prompting actually helps
A meta-analysis of over 100 papers found CoT's gains concentrated in math and symbolic tasks. On MMLU, 95% of the improvement rides on questions containing an equals sign.
On this page
- What does chain-of-thought prompting help with?
- Does chain-of-thought improve every benchmark?
- Why does the equals sign matter on MMLU?
- When should a system route a task to chain-of-thought?
- Is chain-of-thought better than a symbolic solver?
- Are reasoning-trained models pointless if chain-of-thought is selective?
Terms, definedthe jargon, decoded
- Chain-of-thought
- Prompting a model to produce intermediate steps before its answer, rather than emitting the answer directly.
- Symbolic reasoning
- Manipulating formal elements according to rules, where the intermediate steps are the work rather than an explanation of it.
- Symbolic solver
- Purpose-built software that applies formal rules or performs a defined calculation, as opposed to a model predicting the next token.
- Inference cost
- What you pay per request, which reasoning tokens increase whether or not they change the answer.
- Selective application
- Routing only the requests that need a technique to it, instead of enabling it globally.
Chain-of-thought prompting is often treated as a general intelligence switch. Add an instruction like "think step by step," and the model appears to become more capable across the board. The evidence supports a narrower conclusion: the gains are concentrated, not general.
Prompted reasoning is an inference strategy, not a universal quality setting. If a task does not require the model to transform symbols, calculate, or maintain a logical chain, extra reasoning tokens become a recurring cost with little measurable return.
What does chain-of-thought prompting help with?
Symbolic reasoning, mathematics, and logical reasoning. Sprague et al. analyzed a meta-dataset covering over 100 papers and ran their own experiments across 20 datasets and 14 models. The three categories that benefited most were symbolic reasoning, math, and logical reasoning, with average improvements of 14.2, 12.3, and 6.9 respectively.
What those categories share is that intermediate steps are the work rather than a narration of it. Symbolic reasoning asks a model to manipulate formal elements by rule. Mathematics requires intermediate calculation. Logical reasoning requires preserving relations across several deductions. A useful chain gives the model room to carry out those operations instead of committing to an answer immediately.
The same paper identifies the limit of that mechanism: much of the gain comes from improving symbolic execution, while chain-of-thought underperforms relative to using a symbolic solver. A model can explain a calculation and still be a worse calculator than software built to perform it.
Does chain-of-thought improve every benchmark?
No. Across the study's other task categories, average performance with chain-of-thought was 56.8, compared with 56.1 without it.
That pair is the practical warning. It does not establish that every individual task is unaffected; it shows that a broad default can conceal a small average return outside the categories where stepwise execution is central. Paying for additional inference on every request is a poor substitute for measuring which requests need it.
Knowledge retrieval, classification, and direct transformation all contain difficult cases, but difficulty alone does not prove that a visible reasoning trace will help. The relevant question is narrower: must the model execute intermediate symbols or deductions to produce the answer? If not, chain-of-thought deserves a test rather than an automatic place in the prompt.
Why does the equals sign matter on MMLU?
Because it shows a broad benchmark hiding a narrow source of benefit. Sprague et al. report that as much as 95% of the total performance gain from chain-of-thought on MMLU is attributed to questions containing = in the question or the generated output.
An equals sign is not itself a theory of reasoning. In this analysis it simply identifies questions and outputs associated with arithmetic and symbolic manipulation. What the result changes is the interpretation of the headline number: most of the apparent improvement is concentrated in a subset where the model is doing calculation-like work.
"Chain-of-thought improves MMLU" is therefore incomplete as an engineering conclusion. "The measured MMLU gain was concentrated in examples containing an equals sign" suggests both a routing feature and a testable hypothesis, since a system can separate calculation-like requests from those needing only a direct answer.
When should a system route a task to chain-of-thought?
When the task requires symbolic manipulation the model must execute step by step — and as a measured policy rather than a permanent assumption about every hard prompt.
| Task category | What chain-of-thought buys | Better default decision |
|---|---|---|
| Symbolic reasoning | The largest measured category gain, through improved symbolic execution | Test it, then compare against a symbolic solver |
| Mathematics | A large average gain among the studied categories | Route calculation-heavy cases to reasoning and validate outputs |
| Logical reasoning | A positive average improvement from intermediate deductions | Measure on the task's actual logic patterns |
| Other studied categories | 56.8 with chain-of-thought versus 56.1 without, on average | Do not enable globally without task-level evidence |
A practical evaluation set groups requests by the operation they require: symbolic transformation, calculation, logical deduction, factual response, classification, or direct text transformation. Compare the same model and task set with and without prompted reasoning, and keep quality and inference cost as separate measurements.
The routing decision is not "easy prompt versus hard prompt." A short arithmetic request can need a reasoning path; a long factual request can need none. The operation the task requires is a better first classifier than the number of words in the prompt.
Is chain-of-thought better than a symbolic solver?
Not automatically, and for symbolic work that comparison is the one worth running. Sprague et al. report that chain-of-thought improves symbolic execution but underperforms relative to using a symbolic solver.
A language model can infer which operation a task describes, produce an intermediate sequence, and state a result. A solver is built to apply formal rules or perform a defined calculation. When the task fits the solver's formalism, the solver is the appropriate comparison. Chain-of-thought remains useful when a request combines interpretation with reasoning, or when no suitable solver covers the task.
Visibility is a separate matter from correctness. Chain-of-thought can make intermediate work legible, but legibility is not verification, which is why a reasoning trace is not an audit log. Output format adds another variable: if an application also requests structured output, the format constraint and the reasoning strategy should be evaluated together rather than assumed harmless, a concern covered in does JSON mode make your model worse at reasoning.
Are reasoning-trained models pointless if chain-of-thought is selective?
No, and it is worth stating the limit of the finding plainly. It concerns prompted chain-of-thought in the studied setting, not the value of reasoning-trained models as a class. It does not establish that reasoning should be disabled on an unmeasured task, and it does not show that a model's internal reasoning behavior has the same effect as adding a chain-of-thought instruction to a prompt.
The supported claim is narrower: a prompt-level reasoning method should not become a global default merely because it helps some benchmarks. Measure the task, the model, output quality, and inference cost. A reasoning-trained model may still be the right choice, and this paper offers no evidence for turning reasoning off without an evaluation.
For agentic systems, the decision belongs in the task router and the cost model, alongside the broader question of what an agent costs to run. The paper's lesson is not that chain-of-thought is useless. Its value is concentrated, and treating it as a routing decision preserves the cases where it helps while removing the cost of assuming every answer needs an extended reasoning process.
When does chain-of-thought prompting help?
Chain-of-thought helps most when a task requires symbolic reasoning, mathematics, or logical reasoning. Sprague et al. found the largest average improvements in those three categories. The method should still be evaluated on the specific task, because the result does not establish a benefit for every prompt within a category.
Does chain-of-thought help with MMLU?
Chain-of-thought produced a measured gain on MMLU, but Sprague et al. report that as much as 95% of the total gain was attributed to questions containing = in the question or generated output. The improvement was concentrated in arithmetic and symbolic examples rather than distributed across all knowledge tasks.
Should chain-of-thought be enabled for every prompt?
Not without measurement. In the study's other task categories, average performance was 56.8 with chain-of-thought and 56.1 without it. That comparison supports selective routing, while leaving open the possibility that particular unmeasured tasks benefit from reasoning.
Is a symbolic solver better than chain-of-thought?
For work that fits a solver's formalism, often yes. Sprague et al. report that chain-of-thought improves symbolic execution but underperforms relative to using a symbolic solver. Teams should compare both paths on the operations their system actually needs to perform.
Does this mean reasoning models are not worth using?
No. The finding is about prompted chain-of-thought in the studied setting, not about reasoning-trained models as a class. It gives no evidence for disabling reasoning on a task you have not measured; it argues against making a prompt-level technique a global default without task-level evidence.