Reasoning Technologies — in Modern AI

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Foundations: What Is Reasoning in an LLM?
Prompting-Based Reasoning
Self-Consistency and Ensembling
Search-Based Reasoning
Verifier-Guided Reasoning: ORMs and PRMs
Test-Time Compute Scaling
RL-Trained Reasoning: o1, R1, GRPO
Reasoning via Distillation
Tool Use as Reasoning
Agentic Reasoning
Multi-Agent Debate and Critique
Reasoning Calibration and Uncertainty
Math Reasoning
Code Reasoning
Visual / Multimodal Reasoning
Long-Context Reasoning
Reasoning Failures and Pitfalls
Reasoning Safety and Interpretability
Hybrid Reasoning Systems
Architectural Innovations for Reasoning
Practical Recipes
Production Stack: 2026 Defaults

Appendix A: Twenty-Five Things to Memorize

Appendix B: Decision Tree — "Which Reasoning Method?"

Appendix C: Year-by-Year Reasoning Milestones

1. Foundations: What Is Reasoning in an LLM?

1.1 The working definition

"Reasoning" in modern AI = generating intermediate computational steps that mediate between input and output, making complex problems solvable. Not symbolic deduction (though it overlaps); a continuum from pattern completion to multi-step search.

1.2 System 1 vs System 2

System 1 (fast): one-shot answer from base model. System 2 (slow): deliberate, multi-step, sometimes search-augmented. Modern frontier models implement both, choosing dynamically.

1.3 The three eras

2020–2022 (Prompting era): Chain-of-Thought, Self-Consistency, Tree-of-Thought. Reasoning emerges from prompts.
2023–2024 (Search + Verifier era): MCTS + Process Reward Models. Reasoning improves with test-time compute.
2024–2026 (RL-trained era): o1, R1, QwQ. Reasoning is trained into model weights via RL with verifiable rewards. Test-time compute and trained reasoning compose.

1.4 The reasoning gap

Some tasks require many serial steps (math derivations, multi-hop QA). Pre-training alone — which optimizes next-token likelihood — doesn't reliably teach the model when and how to deliberate. Reasoning techniques close this gap.

1.5 Why this matters

At fixed parameter budget above \(\sim 7\) B, scaling test-time compute or RL on reasoning reliably outperforms scaling parameters or pretraining tokens for math, code, and reasoning tasks. This has reshuffled the entire scaling roadmap.

Key

The 2025 lesson: compute spent at inference is now a first-class scaling axis alongside parameters and pretraining tokens. Reasoning techniques convert inference compute into accuracy.

2. Prompting-Based Reasoning

2.1 Zero-shot Chain-of-Thought (Kojima et al. 2022)

Append "Let's think step by step" to the prompt. The model emits a reasoning chain before the answer. Free, surprisingly effective.

2.2 Few-shot Chain-of-Thought (Wei et al. 2022)

Provide a few exemplars of (question → reasoning → answer) before the actual query. Stronger than zero-shot when exemplars match the query distribution.

2.3 Self-Consistency (Wang et al. 2022)

Sample \(K\) chains-of-thought at temperature \(T > 0\); majority-vote the final answer:

\[\hat{y} = \arg\max_y \sum_{k=1}^{K} \mathbb{1}[y^{(k)} = y].\]

Cheap, robust; +5–15 absolute points on math benchmarks. Standard in any reasoning prompt.

2.4 Tree-of-Thought (ToT, Yao et al. 2023)

Frame reasoning as tree search:

Decompose the problem into intermediate "thoughts."
Branch: generate \(K\) candidate next thoughts.
Evaluate each via an LLM-as-judge or heuristic.
Search via BFS or DFS to a depth budget.

Outperforms CoT on Game-of-24, crosswords, creative writing. Needs \(\sim 10\)–\(100\times\) inference cost.

2.5 Graph-of-Thought (GoT, Besta et al. 2024)

Generalize ToT to a DAG: thoughts can be merged, refined, looped. Useful when sub-problems have shared structure.

2.6 Skeleton-of-Thought

Two-stage: model first generates an outline (skeleton), then expands each section in parallel. Speeds up generation and improves structure.

2.7 Plan-and-Solve

First produce an explicit plan, then execute. Stronger than CoT on multi-step problems where planning errors propagate.

2.8 Least-to-Most prompting

Decompose problem into sub-problems ordered easiest-to-hardest; solve each with the previous answers in context.

Strong on compositional generalization.

2.9 Self-Refine, Self-Critique, Reflexion

Self-Refine (Madaan et al.): generate → critique own output → revise. Loop \(K\) times.
Self-Critique: same idea with explicit critique prompts.
Reflexion (Shinn et al.): episode-level reflection on failures stored in episodic memory; subsequent episodes use it.

Effective when the task has verifiable feedback (test cases, rubrics).

2.10 Step-Back Prompting (Zheng et al. 2023)

Ask the model to first articulate higher-level concepts / general principles, then apply. Improves reasoning by \(\sim 5\)–\(10\) points.

2.11 Analogical / similarity-based prompting

Retrieve similar solved problems from a knowledge base; include them as exemplars. "Analogical Prompting" generates exemplars on-the-fly.

2.12 Faithful CoT

Explicit interpretable reasoning steps (Python, logic programs, equations) that can be checked. Reduces hallucination by binding the answer to verifiable computation.

2.13 Decomposition + Verification

Decompose: break into sub-questions.
Solve sub-questions independently.
Verify each: classifier or rule-based.
Aggregate: combine sub-answers.

3. Self-Consistency and Ensembling

3.1 Standard self-consistency

Sample \(K\) chains at \(T > 0\); majority vote answers. \(K = 5\)–\(40\) typical.

3.2 Universal Self-Consistency (Chen et al.)

For free-form answers (no closed-set), use an LLM to compare and select the most consistent. Generalizes vote to non-categorical outputs.

3.3 Verify-then-vote

Filter chains with self-evaluator; vote only over verified ones. Reduces noise from incorrect chains.

3.4 Weighted majority vote

Weight by per-chain confidence (token entropy, RM score, PRM aggregate). Stronger than naive vote.

3.5 Best-of-N

Sample \(N\) candidates; pick the one with highest reward (RM, PRM, verifier). Expected best-of-N improvement (Gaussian approx):

\[\mathbb{E}\!\left[\max_{i \le N} r_i\right] \approx \mu + \sigma\sqrt{2\ln N}.\]

Diminishing returns; sweet spot \(N = 8\)–\(64\).

3.6 Optimization-time scaling laws (Snell et al. 2024)

For a given task and model:

Easy problems: best-of-N wins (parallel sampling).
Hard problems: search (MCTS/ToT with verifier) wins.
Adaptive selection between regimes can match a \(14\times\) larger model at the same compute.

Key

The takeaway from inference-time scaling research: compute is a substitute for parameters, but the optimal allocation varies by problem difficulty. Adaptive routing matters.

4. Search-Based Reasoning

4.1 Beam search reasoning

Maintain top-\(B\) partial reasoning chains by RM/PRM score; expand the best at each step. Cleaner than naive sampling for tasks with clear partial-quality signal.

4.2 Monte Carlo Tree Search (MCTS) for reasoning

Adapt MCTS from games to reasoning:

Selection: descend tree by UCB on PRM/value scores.
Expansion: sample \(K\) candidate next-steps with policy LM.
Simulation: rollout to terminal answer; verify.
Backpropagation: update node values.

\[\text{UCB1:}\quad a^* = \arg\max_a \; Q(s, a) + c\sqrt{\frac{\ln N(s)}{N(s, a)}}.\]

4.3 rStar / rStar-Math

Two-LM MCTS: a generator policy and a discriminator/critic that scores intermediate steps. Discriminator-confirmed branches preferred. Strong on MATH, AIME.

4.4 Mulberry, Marco-o1, ReST-MCTS∗

Various MCTS-style implementations applied to long-CoT reasoning. Mulberry: collective MCTS with step-level critique. Marco-o1: MCTS combined with online preference data collection.

4.5 AlphaProof, AlphaGeometry

DeepMind's frontier formal-math systems. Combine LLM (informal proof generator) with symbolic solver (Lean tactics, geometric reasoning), MCTS guidance. AlphaGeometry 2 reached IMO gold-medal level on geometry.

4.6 REBASE, Process Reward Search

Process-reward-guided beam search. Each candidate continuation scored by PRM; only top-\(B\) kept.

4.7 When to use search

Verifier exists (math, code).
Solution requires many steps where errors compound.
Compute budget allows \(10\)–\(1000\times\) over single-pass.
Pre-trained model is competent at single steps but struggles to chain them.

4.8 Computational cost

Search inflates inference \(10\)–\(10000\times\). The o1 / o3 era trades inference cost for capability; this is the new economics of frontier AI.

5. Verifier-Guided Reasoning: ORMs and PRMs

5.1 Outcome Reward Models (ORMs)

Score only the final answer. Train via BT pairs (correct vs incorrect):

\[\mathcal{L}_{\text{ORM}} = -\mathbb{E}\big[\log \sigma\big(r_\phi(y_w) - r_\phi(y_l)\big)\big].\]

Cheap to train; loses gradient on the reasoning steps.

5.2 Process Reward Models (PRMs)

Score each intermediate step:

\[r_\phi(s_{\le t}) \in [0, 1] \quad \text{for each } t.\]

Loss with per-step binary labels:

\[\mathcal{L}_{\text{PRM}} = -\sum_t \big[y^t \log p_\phi + (1 - y^t)\log(1 - p_\phi)\big].\]

5.3 PRM800K, Math-Shepherd, OmegaPRM

PRM800K (OpenAI): 800k human-labeled math reasoning steps. Expensive.
Math-Shepherd: auto-label by sampling \(K\) continuations from each prefix; label step as good if \(> \tau\) fraction succeed. Scalable.
OmegaPRM: tree-search-based labeling; assigns step credit via Monte Carlo.

5.4 Implicit PRM (from outcome only)

DPO-style closed-form trick gives:

\[r_\phi(s_{\le t}) = \beta \log \frac{\pi_\phi(y^* \mid s_{\le t})}{\pi_{\text{ref}}(y^* \mid s_{\le t})}.\]

Trained as DPO; recovered as a step-level scorer.

5.5 Generative reward models (LLM-as-judge)

Prompt an LLM: "rate this step's correctness 1–10 with reasoning." Cheap, flexible. Biases: position, length, self-preference. Best practice: pairwise comparison + order-swap + CoT-before-rating.

5.6 Inference-time PRM use

Best-of-N: pick highest-PRM-aggregate response.
Search guidance: use PRM as MCTS heuristic / beam ranker.
Reward-shaping for RL: provide dense per-step signal during GRPO.

5.7 Verifier-as-search-heuristic

Combine PRM (heuristic) with search (MCTS / beam) for the strongest results. rStar-Math and Mulberry are canonical examples.

Key

★ 2026 SOTA update — Online implicit process rewards

PRIME: derives an online-updated implicit PRM from outcome labels only (no per-step human labels), giving dense RL rewards while curbing reward hacking.

6. Test-Time Compute Scaling

6.1 The scaling axis

For a fixed model, accuracy improves with inference compute spent on:

Sampling more responses (best-of-N, self-consistency).
Searching deeper (ToT, MCTS).
Longer chains (more reasoning tokens).
Verifier-guided refinement.

6.2 Snell et al. inference-time scaling laws

A \(14\times\) smaller model with optimal test-time compute matches a larger model at the same total compute. The result: spend more on inference, less on parameters.

6.3 Adaptive compute allocation

Easy queries: short response, no search.
Medium: self-consistency over \(K = 8\).
Hard: MCTS with PRM, \(K \sim 100\).
Use a difficulty estimator (small LLM) to route.

6.4 Pareto-optimal recipes per task

Math: long CoT + self-consistency → MCTS + PRM as compute grows.
Code: best-of-N with test execution as verifier.
Open-ended writing: 1–3 samples + self-refine.
Multi-hop QA: ReAct + retrieval, sometimes ToT.

6.5 Cost vs quality table

Method	Compute multiplier	Typical gain	Best for
Zero-shot	\(1\times\) baseline	baseline
CoT	\(1.5\)–\(3\times\)	+5–15%	math, multi-step
Self-consistency	\(K = 8\), \(8\times\)	+5–15%	math, choose
Best-of-N + RM	\(N\times\)	+5–20%	if RM available
Tree-of-Thought	\(10\)–\(100\times\)	+5–25%	search-friendly
MCTS + PRM	\(100\)–\(1000\times\)	+5–30%	math/code
RL-trained CoT	\(1\)–\(5\times\) inference	+20–50%	permanent improvement

Key

★ 2026 SOTA update — Controlling test-time compute

s1: 1K-example SFT plus 'budget forcing' (append 'Wait' to extend, or truncate to cut) to steer think length; s1-32B beats o1-preview on math.

DeepConf: model-internal confidence to filter/weight parallel reasoning traces online or offline; up to 99.9% AIME25 with ~85% fewer tokens, no training.

7. RL-Trained Reasoning: o1, R1, GRPO

7.1 The paradigm shift

Train the model to spontaneously emit long, deliberate reasoning by rewarding final-answer correctness. The model learns when and how to deliberate; reasoning quality scales with training compute.

7.2 OpenAI o1 / o3 / o4

Closed details. Public claims:

Long hidden CoT (only a summary shown to user).
RL-trained for reasoning quality with verifiable rewards.
Test-time compute is a control knob (more thinking = better answer).
Strong jumps on math (AIME), competition coding (Codeforces), GPQA.

7.3 DeepSeek-R1 / R1-Zero

Open recipe that demonstrated reasoning emerges from pure RL.

R1-Zero: pure RL on a base LLM (DeepSeek-V3-Base) with GRPO + verifiable rewards (math correct/incorrect, code passes test, format adherence). No SFT. Reasoning emerges:

Long visible CoT spontaneously appears.
"Aha moments": model self-corrects mid-reasoning.
Average response length doubles over training.

R1: 4-stage pipeline:

Cold-start SFT on \(\sim\) thousands of curated long-CoT examples.
RL with GRPO (verifiable rewards + language consistency).
Rejection-sampling SFT: collect 600k correct + 200k general SFT, retrain.
Final RL pass for safety + helpfulness + general reasoning.

7.4 The GRPO objective for reasoning

For prompt \(x\), sample \(G\) responses; reward \(r_i\). Group-relative advantage:

\[\hat{A}_i = \frac{r_i - \operatorname{mean}(\{r_j\})}{\operatorname{std}(\{r_j\}) + \epsilon}.\]

Per-token clipped objective (no value head):

\[\mathcal{J}_{\text{GRPO}} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\Big(\rho_{i,t}\,\hat{A}_i,\; \operatorname{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\,\hat{A}_i\Big) - \beta\, D_{\text{KL}}\!\left(\pi_\theta \,\|\, \pi_{\text{ref}}\right)\right],\]

where \(\rho_{i,t} = \dfrac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} \mid x, y_{i,<t})}\).

7.5 Verifiable rewards used

Math: regex on \boxed{...} + sympy equivalence.
Code: sandboxed test execution; pass-rate or all-pass.
Format: regex on <think>...</think><answer>...</answer>.
Length penalty: mild, prevent unbounded scaling.
Language consistency: fraction of tokens in target language.

7.6 Open replications and extensions

TinyZero: minimal R1-Zero on small models (showing it works at 0.5B with simple tasks).
Open-R1 (HuggingFace): full R1 reproduction in open code.
SimpleRL, Logic-RL, ReST-MCTS∗.
QwQ, Qwen3-Reasoner: Alibaba's open reasoning line.
Marco-o1, Skywork-o1: o1-style with MCTS at training time.
Mulberry: collective MCTS with step-level critique.

7.7 Multimodal R1-style

Same machinery on VLMs:

VLM-R1, Vision-R1, R1-V: programmatic visual rewards (IoU, mask-IoU, EM).
MM-EUREKA: math + visual + format composite reward.
LMM-R1: multimodal reasoning with cross-modal rewards.
Video-R1: extends to video VLMs.

Key

The R1 era proved: verifiable rewards + GRPO + base LLM \(\Rightarrow\) emergent reasoning, replicable in open. The open-vs-closed gap on reasoning narrowed dramatically in 2025.

Key

★ 2026 SOTA update — New RLVR algorithms & systems

DeepSeek-R1 (Nature): peer-reviewed version confirming pure GRPO RL incentivizes emergent reasoning (self-reflection, aha moments) without human reasoning traces.

DAPO: fully open-sourced large-scale RL system (Decoupled-clip + Dynamic sampling, verl); ~50 on AIME24 from Qwen2.5-32B, all training details released.

Kimi k1.5: long-context RL scaling recipe that drops MCTS/value-functions/PRMs; strong long-CoT plus short-CoT (long2short) distillation.

Absolute Zero: self-play RLVR with zero external data — one model proposes and solves its own tasks, verified by a code executor.

8. Reasoning via Distillation

8.1 The pattern

Generate long-CoT traces with a strong reasoner (R1, o1); SFT a smaller model on those traces. The smaller model inherits reasoning patterns at fraction of inference cost.

8.2 R1 distillation (DeepSeek)

DeepSeek released R1-Distill series: Qwen 1.5B/7B/14B/32B and Llama 8B/70B fine-tuned on \(\sim\) 800k R1 traces.

Strong reasoning quality at far smaller scale; outperformed many larger non-reasoning models on math.

8.3 Cold-start for downstream RL

Distilled traces serve as the SFT cold-start before further GRPO. Stabilizes RL and accelerates convergence.

8.4 Trace quality matters

Filter for correct final answers.
Prefer concise but complete chains over rambling.
Diversity: multiple solution styles per problem.
Reject reward-hacked patterns.

8.5 Distillation vs direct RL

Distillation: cheap, fast, leverages a frontier teacher; bounded by teacher quality.
Direct RL: more expensive, but the student can surpass any specific teacher; required if no strong teacher exists.

In practice: distill to bootstrap, then RL to improve beyond teacher.

Key

★ 2026 SOTA update — Data-efficient reasoning SFT

LIMO: ~817 curated long-CoT examples elicit strong math reasoning (63% AIME24, 95.6% MATH500), challenging the massive-data assumption.

s1K: 1,000-sample reasoning set selected for difficulty/diversity/quality that, with SFT, unlocks competitive test-time-scaling behavior.

9. Tool Use as Reasoning

9.1 ReAct (Reason + Act, Yao et al. 2022)

Interleave reasoning steps and tool calls:

Thought: I need to compute the price.
Action: calculator(34 * 17)
Observation: 578
Thought: That's the answer.
Answer: 578

Strong baseline for tool-augmented reasoning.

9.2 Toolformer (Schick et al.)

Self-supervised: model decides when to call tools by inserting tool-call tokens; trained via filtering helpful calls.

Inline tool integration without explicit prompting.

9.3 PAL (Program-Aided Language Models)

Generate Python code as the reasoning trace; execute it for the final answer. Removes arithmetic errors.

9.4 Program-of-Thoughts (PoT)

Same idea: generate executable program; final answer comes from execution. Strong on math benchmarks.

9.5 Code Interpreter / Code Sandbox

A general-purpose tool: run model-generated Python in a sandbox, return output. Used by GPT-4 Code Interpreter, Claude Code Execution, Gemini.

9.6 Retrieval as a tool

Model decides when to retrieve from a knowledge base / web; uses retrieved content as context. Used by Self-Ask, ReAct, agentic systems.

9.7 Browser as a tool

Browse-the-web actions: navigate, click, scroll, type. Used by Claude Computer Use, OpenAI's Operator, Gemini's browsing, OS-Atlas, ShowUI, UI-TARS.

9.8 API / function calling

Schema-validated structured tool calls. JSON-mode + function calling has become a standard model capability (GPT-4, Claude, Gemini, all open frontier models).

9.9 When code is the reasoning trace

For numerical / algorithmic tasks, generating code is often more reliable than English CoT:

Symbolic computation (sympy).
Combinatorial search (Python loops).
Algorithm execution with print statements.
Verification by execution.

Key

★ 2026 SOTA update — RL for tool-integrated reasoning

Search-R1: RL trains the model to interleave multi-turn search-engine queries inside its reasoning for retrieval-augmented QA.

ReTool: RL teaches strategic code-interpreter invocation within long-CoT; large AIME gains over text-only RL (72.5% at 32B).

R1-Searcher: two-stage outcome-based RL that incentivizes autonomous search/retrieval during reasoning without SFT priors.

10. Agentic Reasoning

10.1 The agentic loop

Sense → Plan → Act → Observe → Reflect → Update. Iterate until task complete or budget exhausted.

Generalization of ReAct to multi-tool, long-horizon tasks.

10.2 AutoGPT, BabyAGI patterns

Single LLM as planner + executor in a loop. Maintains: task list, memory, sub-agent calls. Often brittle because of error accumulation.

10.3 Reflexion (Shinn et al.)

Episode-level reflection on failure stored in memory; subsequent episodes use it to avoid repeating mistakes.

Effective with verifiable feedback.

10.4 Voyager (Wang et al.)

LLM agent in Minecraft with auto-curriculum + skill library + iterative skill acquisition. Demonstrated open-ended skill acquisition.

10.5 Plan-Act-Reflect frameworks

Plan: decompose task into subgoals.
Act: execute one subgoal via tools.
Reflect: evaluate progress; revise plan if needed.
Repeat.

10.6 Modern agentic systems (2025–2026)

Claude Agent SDK / Computer Use: Anthropic's agent framework with screen + bash + file tools.
OpenAI Operator: autonomous web agent.
Devin (Cognition): software engineering agent.
Cursor / Aider / Cline: code agents in IDEs.
LangChain / LangGraph: framework for building agentic flows.
CrewAI, AutoGen, OpenAI Swarm: multi-agent orchestration.

10.7 HRM (Hierarchical Reasoning Model)

Two-network architecture: slow planner produces high-level subgoals; fast executor handles each subgoal. Mirrors System-1/System-2 split in architecture rather than prompting.

10.8 Generative agents (Park et al.)

Agents with memory streams + reflection + planning. Behave like NPCs with personalities. Used in social simulation studies.

10.9 Failure modes

Doom loops: agent retries the same failed action.
Context explosion: history grows unboundedly.
Tool-call hallucination: invented tool names.
Off-task wandering: forgets the original goal.
Mitigations: explicit budget, periodic re-plan, summarization.

11. Multi-Agent Debate and Critique

11.1 Multi-Agent Debate (Du et al. 2023)

Multiple LLMs propose answers; iteratively critique and revise based on others' answers. Convergence to consensus often beats single-agent.

11.2 LLM-as-judge / panel

Multiple LLMs judge a candidate; majority vote or weighted aggregation. Higher reliability than single judge; debiases position / self-preference effects.

11.3 Adversarial / red-team agents

One agent tries to find flaws in another's reasoning. Useful for safety + correctness.

11.4 Society of Mind

Specialized agents (planner, coder, debugger, reviewer) collaborate. Used in MetaGPT, ChatDev. Effective for software-engineering tasks.

11.5 Constitutional debate

Each agent argues from a constitution / principle; debate evaluated against the principles. Used in alignment work.

12. Reasoning Calibration and Uncertainty

12.1 Sampling temperature and top-p

\(T = 0\): deterministic; misses self-consistency benefits.
\(T = 0.6\)–\(0.8\): typical reasoning sampling.
\(T > 1\): more diversity, more incoherence.
top-p (\(\sim 0.95\)): nucleus sampling truncates tail.

12.2 Verbalized confidence

Ask the model to estimate its own confidence. Often miscalibrated (especially overconfident); recalibrate via Platt scaling on held-out data.

12.3 Self-consistency as uncertainty

The fraction of \(K\) samples that agree on the same answer is an empirical confidence:

\[\hat{p}(\text{correct}) \approx \frac{\#(\text{majority answer})}{K}.\]

Useful for routing (high-confidence: answer; low: defer to search/human).

12.4 Token-level entropy

Per-token entropy of the next-token distribution. High entropy at branching points indicates uncertainty. Used in adaptive sampling and to detect hallucination.

12.5 Conformal prediction

Statistical guarantee on coverage by predicting a set rather than a point. Adapts to LLMs via per-token nonconformity scores.

13. Math Reasoning

13.1 Benchmarks

GSM8K: grade school word problems; saturated.
MATH: competition math; mostly solved by frontier.
AIME: USA olympiad-qualifier problems.
HMMT, USAMO, IMO: harder; open frontier.
MathBench, OlympiadBench, Omni-MATH.

13.2 Specialized math models

DeepSeek-Math, DeepSeek-Coder-V2-Math: open math-RL'd line.
Qwen2.5-Math, Qwen3-Math.
Llemma, MetaMath, WizardMath, MAmmoTH: earlier specialized models.

13.3 Formal proof systems

Lean 4: Microsoft / Mathematicians' formal proof language.
Coq, Isabelle/HOL, Agda: alternatives.
LeanInteract, miniF2F: LLM-Lean interfaces.
LeanDojo, ProofNet: training environments.

13.4 AlphaProof, AlphaGeometry

DeepMind's frontier formal-math systems. AlphaGeometry-2 reached IMO gold-medal level on geometry; AlphaProof solved problems on IMO 2024 / 2025.

13.5 Code-augmented math

Generate Python (sympy) for symbolic manipulation, numerical verification, exhaustive search.

Often more reliable than English-CoT alone.

Key

★ 2026 SOTA update — Open formal-proof provers

DeepSeek-Prover-V2: RL + recursive subgoal decomposition in Lean 4 (CoT and non-CoT modes); 88.9% MiniF2F-test, an open alternative to AlphaProof.

14. Code Reasoning

14.1 Benchmarks

HumanEval, MBPP: legacy; saturated.
LiveCodeBench: continually updated to avoid contamination.
SWE-bench / SWE-bench-Verified: real GitHub issues.
Codeforces / IOI / USACO: competitive programming.
BigCodeBench, ClassEval, CRUXEval, RepoBench.

14.2 Strong models (2026)

Closed: GPT-5 / Claude Opus 4.6 / Gemini 2.5.
Open: DeepSeek-R1, DeepSeek-Coder-V2, Qwen-Coder, Yi-Coder.

14.3 Test-execution as verifier

Generate code; run unit tests in sandbox; reward = pass-rate. Standard for code RL fine-tuning.

14.4 Iterative code-and-fix loops

Generate → test → fix loop. Standard in modern coding agents (Aider, Cursor, Cline, Devin).

14.5 Software-engineering agents

Devin, Codex Cloud: commercial SWE agents.
SWE-agent: open framework with structured action space (file, edit, search).
SWE-Gym: training environment for SWE agents.

15. Visual / Multimodal Reasoning

15.1 Visual chain-of-thought

Encourage VLM to verbalize what it sees, reason about it, then answer. Surprisingly effective; +5–15 points on visual reasoning benchmarks.

15.2 LLaVA-CoT, LLaVA-o1

Structured stages: summary → caption → reasoning → conclusion. MCTS at inference for harder problems.

15.3 Mulberry, Insight-V

Collective MCTS for VLMs; step-level critique; multi-pass reasoning with vision.

15.4 Vision-R1, VLM-R1, MM-EUREKA, R1-V

GRPO + verifiable visual rewards (IoU, mask-IoU, exact-match QA, format) on VLMs. R1 paradigm extended to multimodal.

15.5 Spatial reasoning

SpatialVLM, SpatialBot, RoboPoint: train VLMs with synthetic 3D-grounded spatial QA. Improves spatial reasoning for robots / AR.

15.6 Diagram / chart reasoning

ChartQA, MathVista, ChartBench: benchmarks.
Chart-aware tokenization (separate axes / values).
Code-as-tool: generate matplotlib code as the reasoning trace.

15.7 Video reasoning

Video-R1, Video-CoT: long-CoT over video frames. Token-budget management is the bottleneck. Memory mechanisms (MovieChat, MA-LMM) help.

15.8 Visual MCTS

Combine VLM policy + visual PRM + MCTS. Strong on visual math (e.g., MathVista) and visual logic puzzles.

16. Long-Context Reasoning

16.1 Long-document QA

Single document, \(\sim 10\) K–\(100\)K tokens, multiple questions. Needs precise retrieval + reasoning over retrieved spans.

16.2 Multi-document synthesis

Multiple sources; synthesize a coherent answer. Patterns:

Map: extract relevant span per source.
Reduce: combine into single answer.
Or: long-context retrieval + single-pass synthesis.

16.3 Multi-hop reasoning

Answer requires chaining \(\ge 2\) facts from different sources:

Retrieval-then-read with multi-step retrieval.
Self-Ask: model decomposes into sub-questions.
HotpotQA, MusiQue, Bamboogle benchmarks.

16.4 Long-video reasoning

Hour-long video QA. Frame-budget compression + selective attention via question-conditioned retrieval. Standard in modern video VLMs.

16.5 RAG vs long-context (revisited)

Long-context wins for cohesive reasoning.
RAG wins for huge corpora and freshness.
Hybrid: RAG to fetch candidate docs; long-context to reason.

17. Reasoning Failures and Pitfalls

17.1 Memorization vs reasoning

A model can pass GSM8K via memorized solutions. Counter:

Use uncontaminated benchmarks (LiveCodeBench, AIME-newer).
Apply perturbations (rename variables, change numbers).
Test on adversarial reasoning sets.

17.2 Reasoning illusion

Long CoT that looks impressive but doesn't actually drive the answer (the model would've answered the same without it). Test by ablating the CoT.

17.3 Length explosion

Reasoning gets unboundedly long without quality gains. Mitigations: length penalty, Dr. GRPO, explicit cap.

17.4 Format breakdown

At long context, model forgets to wrap the answer in expected format. Mitigations: format-reward in RL, strict template enforcement.

17.5 Self-consistency biases

Most-frequent answer isn't always correct; can amplify systematic errors. Best with verifier-filtered consistency.

17.6 Cognitive overload

Too many reasoning constraints in prompt \(\Rightarrow\) model drops some. Keep prompts focused; chain rather than stack.

17.7 Faithfulness gap

The CoT is post-hoc rationalization, not the actual computation. Implications for safety / interpretability: visible reasoning can be misleading.

Watch out

Don't trust a model's reasoning trace as proof of its underlying computation. Use external verification (test execution, formal proof, retrieval citations) for high-stakes outputs.

Key

★ 2026 SOTA update — Efficient / anti-overthinking reasoning

Chain of Draft: minimal-draft intermediate steps (~5 words each) that match CoT accuracy while using as little as 7.6% of the tokens, cutting latency and cost.

18. Reasoning Safety and Interpretability

18.1 Visible vs hidden CoT

Visible (R1, QwQ): user sees the reasoning. Useful for trust + debugging; risk of misuse / extraction.
Hidden (o1): model reasons internally; only summary shown. Less interpretable but cheaper UX.
Streaming: show CoT as it's generated; mix of both worlds.

18.2 Reasoning audits

Probe reasoning traces for harmful intent / planning.
Detect jailbreak attempts hidden in CoT.
Use a judge model to flag suspicious reasoning.

18.3 Adversarial reasoning

Adversaries can:

Inject malicious sub-goals into CoT prompts.
Use reasoning to bypass safety (multi-step planning).
Exploit token-level format to smuggle harmful content.

Counters: per-step safety filters, constitutional checks, sandboxed tool use.

18.4 Faithfulness studies (Anthropic, OpenAI)

Empirical findings: CoT often not faithful to underlying computation, even for correct answers. Implication: you can't audit a model purely by reading its CoT.

Key

★ 2026 SOTA update — CoT monitoring for safety

Chain of Thought Monitorability: multi-lab position paper arguing readable CoT is a fragile-but-valuable oversight signal, urging developers to preserve it.

19. Hybrid Reasoning Systems

19.1 LLM + classical solver

Hand off structured sub-problems to specialized solvers:

Linear programming: gurobi / OR-tools.
SAT/SMT: Z3.
Symbolic math: SymPy, Mathematica, Lean.
Planning: PDDL planners.
Linear algebra: numpy / SciPy.

19.2 LLM + retrieval

Use embeddings + reranker to fetch facts; LLM synthesizes. Standard RAG pattern.

19.3 LLM + database / SQL

Schema-aware LLM generates SQL; run; iterate on errors. Used in analytics agents.

19.4 LLM + world model

LLM proposes plans; world model (Dreamer / Sora-like) simulates outcomes; LLM revises. Emerging in robotics / AV.

19.5 LLM + code execution

Code as tool; LLM-generated Python in sandbox. Strong for math, data analysis, plotting.

19.6 Mixture of solvers

Route by problem type: math → symbolic; data → SQL; planning → classical planner; open-ended → LLM.

20. Architectural Innovations for Reasoning

20.1 Mixture of Reasoners

Multiple specialized expert models (math, code, general); router selects per query. Used in some commercial agent stacks.

20.2 Hierarchical Reasoning Models (HRM)

Two-net: high-level slow planner + low-level fast executor. Architectural System-1/System-2.

20.3 Reasoning-aware MoE routing

Standard MoE layers, but routing influenced by problem difficulty / domain. Some experts naturally specialize for math vs code vs writing.

20.4 Long-context architectures for reasoning

Reasoning needs huge effective context (CoT + retrieval + tool outputs). Modern models use:

Ring attention for \(> 100\)K context.
RoPE + YaRN for context extension.
MLA / GQA for KV cache efficiency.
Sliding window + sink for streaming.

20.5 Native multimodal reasoning

Models that natively handle text + image + video + audio in one stack (Gemini 2.5, GPT-5, Claude Opus 4.6) avoid the brittleness of bolted-on adapters for cross-modal reasoning.

Key

★ 2026 SOTA update — Latent / continuous reasoning

Coconut: reason in continuous latent space by feeding the last hidden state back as the next input embedding, enabling BFS-like search without token CoT.

Huginn (recurrent-depth): iterate a shared latent transformer block at test time to scale compute in depth without emitting reasoning tokens or growing context.

21. Practical Recipes

21.1 Building a reasoning system from scratch

Key

Reasoning system recipe.

Pick a strong base model (\(\ge 7\) B for emergent reasoning).

Cold-start SFT on \(\sim 10\) k long-CoT exemplars (or distill from R1/o1).

Identify a verifiable reward (math, code, format, programmatic check).

GRPO with \(G = 16\), low \(\beta\), dynamic sampling.

Add inference-time scaling: self-consistency → MCTS for hardest queries.

Combine with tools (calculator, code, retrieval).

Eval: AIME, LiveCodeBench, GPQA, MMLU-Pro, ARC-AGI.

21.2 Cheap reasoning at small scale

Use a distilled R1 / o1 model (R1-Distill-Qwen-7B, etc.).
Self-consistency \(K = 5\)–\(8\).
Tool augmentation (code interpreter for math).
Best-of-N with PRM if available.

21.3 Reasoning over your own data

Strong RAG pipeline (BGE / E5 / SigLIP embedder + reranker).
Long-context VLM for cohesive synthesis.
Code interpreter for any numerical analysis.
Self-consistency + verbalized confidence for routing.

21.4 Reasoning agents in production

Strict tool schema + function calling.
Budget caps (max steps, max tokens).
Periodic re-plan / summarization.
Reflection on failures, persistent memory across sessions.
Human escalation on low-confidence / safety triggers.

21.5 Cost-quality trade-offs

Cheap: small model + CoT.
Mid: medium model + self-consistency + tools.
High: frontier model + MCTS + PRM + tools + multi-agent debate.
Adaptive routing: difficulty estimator picks tier per query.

22. Production Stack: 2026 Defaults

Use case	Default reasoning approach	Notes
General LLM Q&A	CoT prompt + light self-consistency	Cheap, broadly improves
Math (cheap)	R1-Distill + CoT + sympy code	Tools dramatically help
Math (frontier)	GPT-5 / o3 / Claude Opus / R1	Hidden CoT + RL-trained
Code generation	Strong base + iterative test/fix	SWE-agent patterns
Multi-step planning	Plan-and-Solve + verifier checkpoints	Decomposition + verify
Multi-hop QA	ReAct + retrieval + sub-question decomp	Self-Ask pattern
Visual reasoning	VLM + visual CoT + sometimes MCTS	Vision-R1 family
Spatial / 3D reasoning	SpatialVLM-style + 3D-grounded data	For robotics/AR
Agentic tasks	Plan-Act-Reflect + bounded budget	Claude Agent / Devin patterns
Math olympiad / proofs	formal AlphaProof-style: LLM + Lean	Hybrid LLM + symbolic
Long-doc reasoning	Long-context VLM + RAG hybrid	Multi-tier retrieval
Safety-critical	Multi-agent debate + audit + formal	Don't trust visible CoT alone