Chain of Thought
Showing the work
Showing your work helps LLMs on multi-step problems: prompting reasoning before the answer raises accuracy on arithmetic, logic and planning. But reasoning output also leaks, raises latency/tokens, and needs guardrails.
What you will learn
- Prompt step-by-step reasoning correctly
- List CoT's costs and risks
- Decide when to use reasoning at all
What is this?
Chain-of-thought (CoT) asks the model to reason step by step before concluding. For multi-step tasks — arithmetic, logic, planning, extraction chains — CoT measurably lifts accuracy. The costs: more output tokens (→ latency and price), reasoning visible to users, and a 'chain' that can still reason wrongly with confidence.
Why do you need it?
- Single-shot jumps gloss over required intermediate states.
- Forced steps let the model evaluate partial conditions it otherwise skips.
- Reasoning output gives you a weak post-mortem trail.
- Frontier models increasingly do internal reasoning (hidden or visible) — understand both modes.
Mental model
Long division with visible steps
Chain of thought
A student who writes each arithmetic step catches the error before the final line; a student who 'just knows' the answer can't be audited and often misremembers. Unfair to compare the final digit alone — the value is in the work.
How it works
- Ask explicitly: 'reason step by step, then give your final answer.'
- Few-shot CoT: show one worked example with steps.
- Consider 'think carefully, but respond succinctly' to bound verbosity.
- Use structured debug like 'list assumptions first' for planning tasks.
- Measure: some tasks gain nothing; don't pay tokens for them.
CoT flow.
In production
- Complex classification: 'list the signals, then decide.'
- Planning/agent subtasks: 'step through preconditions.'
- Reasoning transcripts aid eval, debugging and audit trails.
Best practices
- Apply CoT to multi-step tasks; skip for trivial/extractive ones.
- Cap reasoning verbosity ('short rationale, then answer').
- Sanitize/limit reasoning exposure for sensitive prompts.
- A/B CoT on/off using your eval set before shipping it.
Common mistakes
- Applying CoT to single-step tasks and paying tokens for no gain.
- Unbounded reasoning digression on chat loops.
- Exposing raw reasoning that contains private context to users.
Common issues
Reasoning overflows and repeatsNo output cap on the chain
Cap max tokens; instruct 'answer within N lines'.
CoT doesn't lift accuracyTask already linear/extractive
Benchmark without CoT; don't force reasoning where it adds noise.
Debugging tips
- Reasoning drifts off-topic → constrain with 'relate steps to the inputs'.
- Answer correct but reasoning 'wrong' → verify both; the model can mismatch.
- Latency creep → hidden-reasoning models or unbounded step chains; bound output.
Interview checkpoint
Quick quiz
Chain-of-thought helps most on…
CoT only ever reduces output latency.