Chain of thought is worth 0.7 points outside math and logic

Chain of thought buys 14.2 accuracy points on symbolic reasoning, 12.3 on math and 6.9 on logical reasoning. On everything else it buys 0.7. That is 56.8 with it against 56.1 without, averaged across every other category tested. In the authors’ own words: “As much as 95% of the total performance gain from CoT on MMLU is attributed to questions containing ‘=’ in the question or generated output.”

Those figures come from To CoT or not to CoT? (Sprague et al., arXiv 2409.12183, last revised 7 May 2025): “a quantitative meta-analysis covering over 100 papers using CoT” plus the authors’ own evaluations of “20 datasets across 14 models.”

I published a post recommending chain of thought for “complex reasoning, coding, math, multi-step decision making.” Coding is not a category showing benefit in that meta-analysis, and neither is multi-step decision making. The ratio between two items I listed as equivalent is 12.3 against 0.7, about 17.6x. Correcting that with the numbers attached is worth more than leaving it standing.

Why a prompt fails in a way a better model cannot fix

From 31 October 2025 to 27 April 2026 I ran a publishing bot against this domain. It made 842 Add post: commits. It used gpt-4o-mini with a gpt-4o rewrite pass. Its entire input per article was 450 characters of RSS description, sliced in code. It had AUTO_PR = "false", so nothing was reviewed before it hit main. 180 of the 739 English posts cited a source called “Internal Analysis” that does not exist.

The prompt asked for an authoritative article with statistics and named sources, from 450 characters that contained neither. No model upgrade repairs that. The prompt specified an output the input could not support, and the model did the only thing left: it filled the shape. The lesson runs opposite to the folk version. Prompts matter more than model choice, but not because a clever prompt adds capability. They matter because the prompt is where you decide what is being asked for, and a badly specified ask fails identically at every model quality. The full account of that pipeline is here. The commit counts come from a private repository, so you have my word for them rather than a link.

1. Chain of thought: real, and narrower than advertised

The technique has two origins and most writing conflates them. Wei et al. (arXiv 2201.11903, January 2022) named it, and their method was few-shot: “a few chain of thought demonstrations are provided as exemplars in prompting.” Kojima et al. (arXiv 2205.11916, May 2022) established the zero-shot version and the phrase “Let’s think step by step.” What almost everyone actually does is Kojima’s, under Wei’s name. The original gains are large and real.

Table: Chain-of-thought accuracy on arithmetic benchmarks, as published in the two originating papers. Wei’s figures are read from Table 2 of the full text; deltas and multiples are my arithmetic from those published numbers.

Every benchmark in that table is arithmetic. Wei also stated a scale condition plainly: the technique “does not positively impact performance for small models, and only yields performance gains when used with models of ∼100B parameters.” That caveat has aged out, since anything a reader touches in 2026 clears it. The task-category caveat has not.

Table: What my post claimed chain of thought helps with, against what the Sprague meta-analysis measured.

The floor is also not zero. Liu et al. (arXiv 2410.21333) took six tasks from cognitive psychology where deliberation degrades human performance and found that “in three of these tasks, state-of-the-art models exhibit significant performance drop-offs with CoT (up to 36.3% absolute accuracy for OpenAI o1-preview compared to GPT-4o).” Verbal overshadowing has a machine analogue.

Then the price. Chain of thought spends output tokens, the most expensive class on every sheet I checked. On Anthropic’s current price list, output bills at exactly 5x input on every model in the table, from Haiku 4.5 at $1/$5 to Fable 5 at $10/$50. OpenAI’s sheet does not hold to a single multiple across its line. I am not going to quote rows off it, and the reason is the article’s own subject: that page renders its table from JavaScript, an earlier draft of this paragraph quoted four model-and-price pairs from it that are not on it at all, including two models it does not list, and the error survived one round of checking. What matters for the argument holds without the enumeration anyway. Output is the expensive class on every sheet I looked at, the multiple over input is several times rather than a few percent, and it is not the same multiple everywhere, so the cost of a reasoning preamble is a vendor-specific number you have to read off the current page rather than a rule of thumb you can carry between them. A worked illustration at Claude Sonnet 5’s introductory $10 per million output tokens: a 100-token direct answer costs $0.0010, and the same answer behind a 900-token reasoning preamble costs $0.0100. At 100,000 calls a month, $100 against $1,000. Outside math and logic you are paying that $900 for 0.7 points. My own arithmetic from published list prices, not a bill anyone has received.

Both vendors have now moved. OpenAI’s reasoning best practices says outright: “Avoid chain-of-thought prompts: Since these models perform reasoning internally, prompting them to ’think step by step’ or ’explain your reasoning’ is unnecessary.” Anthropic’s prompting best practices file manual CoT under “Manual chain-of-thought (CoT) prompting as a fallback,” for when thinking is switched off, and add that “a prompt like ’think thoroughly’ often produces better reasoning than a hand-written step-by-step plan. Claude’s reasoning frequently exceeds what a human would prescribe.” One vendor calls the instruction unnecessary, the other has demoted it to a fallback. Neither is where the folk advice still sits.

2. Few-shot: three to five, and they are not teaching

My post said one or two examples. Anthropic’s current guidance says “Include 3–5 examples for best results,” with examples wrapped in <example> tags and required to be “Diverse: Cover edge cases and vary enough that Claude doesn’t pick up unintended patterns.” Take the vendor’s number over mine.

Why diversity is the requirement exposes that the folk mechanism is wrong. I wrote that models follow patterns better than vague instructions. Min et al. (arXiv 2202.12837) tested that and found that randomly replacing the labels in demonstrations barely hurts performance, across 12 models including GPT-3. What carries the effect is the label space, the distribution of the input text, and the format of the sequence. Examples are not teaching the model correct answers. They are declaring the shape and range of an acceptable response, which is why five lookalike examples do less work than three that span the edges. Caveat: Min et al. tested classification and multiple-choice, so do not carry “labels do not matter” into open-ended generation.

Format is a live variable rather than a cosmetic one. Sclar et al. (arXiv 2310.11324) held semantic content fixed, changed only separators, spacing and casing, and report “performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B.” Two things must travel with that number or it misleads: LLaMA-2-13B is a 13B open-weight model from 2023, not a frontier model, and 76 points is the maximum observed spread, not a typical one. Discounted for both, it still says separators and casing are part of the prompt rather than packaging around it.

Order is the failure people miss. Turpin et al. (arXiv 2305.04388) reordered multiple-choice options in few-shot prompts so the correct answer was always “(A)”. Accuracy dropped by as much as 36% across 13 BIG-Bench Hard tasks, tested on GPT-3.5 and Claude 1.0, and the models wrote fluent reasoning that never mentioned the bias steering them. Your example set is an instruction whether you intended it as one or not.

OpenAI’s guidance on reasoning models, again verbatim: “Try zero shot first, then few shot if needed.”

3. Role prompting: register, not correctness

I claimed a role “changes how the model prioritizes information, explains concepts, and structures its answers.” The first and third parts are supported. The implied fourth, that it makes answers more correct, is not.

Anthropic’s docs claim exactly the modest thing and no more: “Setting a role in the system prompt focuses Claude’s behavior and tone for your use case.” No accuracy claim appears. Zheng et al. (arXiv 2311.10054) tested 162 roles across 4 LLM families on 2,410 factual questions and found that “adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added,” and that where gains appear, “the effect of each persona can be largely random.”

A second paper disagrees, and suppressing that would be dishonest. Kong et al. (arXiv 2308.07702) report role-play gains across twelve reasoning benchmarks with ChatGPT, including AQuA moving 53.5% to 63.8% and Last Letter moving 23.8% to 84.2%. Their explanation is the part that settles it for me: role-play “acts as a more effective trigger for the CoT process.” If that is right, role prompting is an oblique way of invoking technique one rather than a third independent technique, which predicts it adds nothing on a model with reasoning already on. Zheng measured factual accuracy and found no reliable gain; Kong measured reasoning tasks and found gains they attribute to reasoning elicitation. A role reliably changes register and unreliably changes correctness, so use it for voice and audience and never for truth.

Table: What each technique costs and what it reliably buys. Prices are Anthropic list prices for Claude Sonnet 5, read 6 August 2026; the effect column is the measured average from the papers cited above.

They have opposite cost structures and my post presented them as interchangeable. Five 200-token examples cost $0.0020 per call at base input price, or $0.0002 on a cache read, a 10x reduction that repays the 1.25x write premium inside a single reuse. Anthropic’s docs reach the same conclusion independently: “caching pays off after just one cache read for the 5-minute duration.” The cheap technique is the one with the most reliable effect.

How to detect this in your own system

Four checks, none needing a benchmark harness.

Look for an equals sign. If your task does not involve symbolic manipulation, arithmetic or formal logic, the Sprague number for it is 0.7 points, and a “think step by step” instruction in a copywriting, summarisation or classification prompt is spend with no documented return.

Ask whether a human expert would do the task better on instinct than on deliberation. Liu et al. found the drops clustered on tasks with exactly that property.

Stop treating reasoning traces as an audit log. Turpin showed models rationalising a bias they never named. Anthropic’s measurement of 3 April 2025 found Claude 3.7 Sonnet mentioned an inserted hint in its reasoning 25% of the time and DeepSeek R1 39%. In a setup where models were rewarded for following deliberately wrong hints, they took the hint “in over 99% of cases” and admitted to it “less than 2% of the time in most of the testing scenarios.” Outcome-based reinforcement learning on an earlier snapshot of Claude 3.7 Sonnet raised faithfulness and then plateaued, at 28% on MMLU and 20% on GPQA. If you are logging chain of thought for compliance, you are logging a plausible story about the computation.

Last, check that the prompt’s demands are proportionate to its input. The bot failed because it asked for sourced statistics from 450 characters that had none, and no technique on this page would have saved it. List every fact the output requires and confirm each is present in the input. Anything absent will be generated.

What I have now that I did not have before

A decision rule instead of a habit. Chain of thought when the task contains symbols, math or formal logic, and not otherwise. Examples in the published range, three to five, chosen for spread rather than similarity, cached. Roles for voice and never for accuracy.

Both vendor pages cited here are written against whatever models are current, so the rule needs re-reading rather than remembering. One trap documented right now: with extended thinking disabled, “Claude Opus 4.5 is particularly sensitive to the word ’think’ and its variants,” and the docs suggest “consider,” “evaluate,” or “reason through” instead. And three techniques is not the set. Schulhoff et al.’s systematic survey catalogues 58 text-based prompting techniques and defines 33 vocabulary terms.

The post I am correcting was not wrong about 2022. It was describing 2022. These techniques were genuinely transformative when published, and the literature that narrowed them arrived later and got far less attention. Reading the primary source is the only defence, and it takes an afternoon.