Chain-of-Thought (CoT) prompting is a prompt engineering technique that dramatically improves the reasoning capabilities of large language models by guiding them to generate intermediate reasoning steps before arriving at a final answer. Instead of asking a model to jump directly to a conclusion, CoT prompting encourages it to “show its work” — decomposing complex problems into a coherent series of logical steps.
The technique was formalized in a landmark 2022 paper by Google researchers (Wei et al.), who demonstrated that simply adding the phrase “Let’s think step by step” to a prompt could improve GPT-3’s accuracy on grade-school math problems from 17.9% to 57.1%. This was a pivotal moment in prompt engineering — proof that how you ask matters as much as what you ask.
For AI engineers, CoT prompting is a foundational tool. It applies to arithmetic reasoning, commonsense reasoning, symbolic manipulation, multi-step planning, and code generation. It is especially critical in production systems where correctness matters — such as financial calculations, legal reasoning, or diagnostic workflows — because the explicit reasoning chain also provides auditability and debuggability.
Large language models generate text one token at a time, each token conditioned on all preceding tokens. When a model is asked to answer a complex question directly, it must compress all reasoning into the probability distribution for the next token — which often leads to errors. CoT prompting solves this by giving the model “scratch space” in the form of intermediate tokens. Each reasoning step becomes part of the context for subsequent steps, allowing the model to build up complex logic incrementally.
CoT prompting works because autoregressive models condition each output token on all previous tokens. By generating intermediate reasoning tokens, the model effectively creates a working memory that it can reference for later steps — turning a single-hop inference into a multi-hop chain.
The original CoT approach uses few-shot examples that demonstrate step-by-step reasoning. You provide the model with 2-5 examples where the problem is followed by an explicit reasoning chain, then the answer. The model learns to emulate this pattern for new inputs.
## Example prompt with few-shot CoT:
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis
balls. Each can has 3 tennis balls. How many tennis balls
does he have now?
A: Let's work through this step by step.
Roger started with 5 balls.
He bought 2 cans × 3 balls per can = 6 balls.
Total: 5 + 6 = 11 tennis balls.
The answer is 11.
Q: The cafeteria had 23 apples. If they used 20 to make
lunch and bought 6 more, how many apples do they have?
A: Let's work through this step by step.
The cafeteria started with 23 apples.
They used 20, so 23 - 20 = 3 apples remaining.
They bought 6 more, so 3 + 6 = 9 apples.
The answer is 9.Kojima et al. (2022) discovered that you don’t even need examples — simply appending "Let's think step by step" to a prompt triggers reasoning behavior in sufficiently large models. This is called zero-shot CoT and is remarkably effective for its simplicity. It works because the phrase activates reasoning patterns the model absorbed during pre-training.
## Zero-shot CoT prompt:
Q: A bat and a ball cost $1.10 in total. The bat costs
$1.00 more than the ball. How much does the ball cost?
A: Let's think step by step.Without CoT, models frequently answer “$0.10” (the intuitive but incorrect answer). With the step-by-step trigger, models correctly work through the algebra: if ball = x, then bat = x + 1.00, so x + (x + 1.00) = 1.10, therefore 2x = 0.10, x = $0.05.
Wang et al. (2022) introduced self-consistency as a powerful enhancement to CoT. Instead of generating a single reasoning chain (greedy decoding), the model generates multiple independent chains at higher temperature, then selects the most frequent final answer by majority vote. This exploits the insight that correct answers can be reached via different reasoning paths, while incorrect answers tend to be random and inconsistent.
Sample N diverse reasoning paths (e.g., N=10 at temperature 0.7), extract the final answer from each, and take the majority vote. This can improve CoT accuracy by 5-15 percentage points on math and reasoning benchmarks, at the cost of N× inference calls.
Tree-of-Thought prompting (Yao et al., 2023) extends CoT from a linear chain into a branching tree structure. At each reasoning step, the model generates multiple candidate “thoughts,” evaluates which are most promising, and can backtrack or explore alternatives. This enables lookahead and deliberate search strategies, making it suitable for problems like creative writing, puzzle-solving, and planning tasks where the optimal path isn’t obvious from the start.
Manually crafting few-shot CoT examples is labor-intensive. Auto-CoT (Zhang et al., 2022) automates this by using the LLM itself to generate reasoning chains for a diverse set of sampled questions. These auto-generated demonstrations are then used as few-shot examples. The key innovation is clustering input questions by diversity to ensure the generated examples cover different reasoning patterns rather than repeating similar logic.
CoT prompting provides the most benefit for tasks that require multi-step reasoning. It is most effective in these scenarios:
CoT prompting adds latency and token cost. For simple factual lookups (“What is the capital of France?”), pattern matching, or classification tasks, the overhead isn’t justified. Research shows CoT primarily benefits models above ~100B parameters — smaller models may generate plausible-sounding but incorrect reasoning chains.
In production AI systems, CoT is commonly implemented in several ways:
<thinking> block that is parsed out before showing the user the final answer. This gives the model reasoning space without cluttering the user experience.reasoning, confidence, answer) for programmatic evaluation and logging.CoT prompting has demonstrated significant improvements across major benchmarks: