Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting is a prompt engineering technique in which a large language model (LLM) writes out intermediate reasoning steps before giving its final answer. Instead of jumping from a question straight to "9", the model writes "23 - 20 = 3, and 3 + 6 = 9" and states the answer last. Jason Wei and colleagues at Google introduced the technique in the 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. With eight worked examples in the prompt, PaLM 540B went from 17.9% to 56.9% on GSM8K, a benchmark of grade-school math word problems, passing the previous best of 55% from a fine-tuned GPT-3 paired with a separately trained verifier.

How Chain-of-Thought Prompting Works#

An LLM generates text one token at a time, and every token passes through the same fixed number of network layers. A question whose answer depends on several steps leaves the model to do all of that work before it writes the first token of the answer. When the model writes the steps out, each intermediate result enters the context window and conditions every later token, so harder problems get more tokens, and more computation, before the answer appears.

Few-shot CoT puts a handful of exemplars in the prompt, each a question followed by a worked solution, and the model imitates that format on the new question, an instance of few-shot learning from examples in the prompt. This exchange from Figure 1 of the Wei et al. paper shows the format. The first answer is the exemplar written into the prompt, and the second is the model's reply:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?
A: The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.

Given the same exemplar reduced to "The answer is 11.", the model replied "The answer is 27." to the cafeteria question.

Zero-shot CoT needs no exemplars. Takeshi Kojima and colleagues found that appending "Let's think step by step" to a question is enough to trigger a written chain: with InstructGPT (text-davinci-002), the phrase raised zero-shot accuracy on the MultiArith benchmark from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7%, as reported in Large Language Models are Zero-Shot Reasoners.

When Chain-of-Thought Helps#

The gains depend on model scale and task. Wei et al. describe CoT as an emergent ability of scale: improvements appeared only in models of roughly 100 billion parameters, while smaller models "produced fluent but illogical chains of thought" and scored below standard prompting. The largest gains came on multi-step problems in arithmetic, commonsense, and symbolic reasoning. On SingleOp, a set of one-step arithmetic problems, the change was negative or very small.

Sampling several chains improves on a single one. In self-consistency, Xuezhi Wang and colleagues sample diverse reasoning paths with nucleus or temperature sampling and keep the answer most paths agree on, which raised GSM8K accuracy by 17.9 percentage points over a single greedy chain. Every extra chain adds generated tokens, so CoT and self-consistency trade inference latency and cost for accuracy.

Vision-language pipelines use the same pattern with image evidence. In Ultralytics Platform Agents, an LLM block receives an image together with upstream YOLO26 detections, so a chain-of-thought prompt there can have a vision-language model work through the detected objects one at a time before giving its verdict on the scene.

Reasoning Models and Faithfulness#

Reasoning models such as OpenAI's o1, released in September 2024, are trained with reinforcement learning to produce a long chain of thought before every answer, so the behavior no longer depends on the prompt. OpenAI's reasoning best practices tell developers to avoid "think step by step" instructions with these models, since such prompts may not help and can sometimes hurt.

A written chain also looks like an explanation of the answer, which makes it tempting to read as explainable AI. Miles Turpin and colleagues showed that it can misrepresent the real cause in Language Models Don't Always Say What They Think: when few-shot exemplars were reordered so the correct option was always "(A)", GPT-3.5 and Claude 1.0 shifted their answers toward "(A)" and wrote chains that justified them without mentioning the ordering. Biasing features like this one cut accuracy by as much as 36% across 13 BIG-Bench Hard tasks. Plausible but wrong chains are one source of hallucination in LLMs.

Chain-of-Thought vs. Related Techniques#

• Prompt chaining: Splits a task across several model calls, with code passing each output to the next prompt. CoT happens inside one call.
• Tree of Thoughts (ToT): Generates several candidate steps at each point, scores them, and searches the resulting tree with backtracking, where CoT follows a single path.
• ReAct prompting: Interleaves reasoning steps with actions such as searches or tool calls, and feeds the results back into the chain.
• Chain-of-Verification (CoVe): Has the model draft an answer, ask itself verification questions, answer them independently, and revise the draft.
• Latent reasoning: Performs intermediate reasoning in the model's hidden states instead of written tokens, so no chain is visible.