When Chain-of-Thought Prompting was introduced, it showed that large language models could solve many difficult reasoning problems simply by thinking step by step before producing an answer.
It was a remarkable breakthrough, but it also exposed an important limitation: What happens if the model's reasoning is wrong?
Even with Chain-of-Thought, a model follows only a single reasoning path. If that path contains a mistake, the final answer is likely to be wrong as well. Better reasoning still depends on getting the first attempt right.
This paper tackles that limitation with an idea inspired by how people solve difficult problems. Rather than trusting the first solution that comes to mind, we often consider several different approaches before deciding which answer is most convincing. The authors asked whether language models could do the same.
Their answer is Self-Consistency: a simple decoding strategy that generates multiple independent reasoning paths and selects the answer that appears most consistently among them. The model itself remains unchanged. There is no additional training, fine-tuning, or supervision. Only the decoding strategy changes.
Despite its simplicity, the approach produced remarkable improvements across arithmetic, common sense, and symbolic reasoning tasks, showing that more reliable reasoning often comes from comparing multiple lines of thought rather than committing to the first one.
This paper became a natural successor to Chain-of-Thought prompting and marked an important shift in LLM research. Instead of making models larger, it showed that substantial gains could come from making better use of the reasoning abilities they already possessed.
Paper Overview
In this review, we'll explore Self-Consistency Improves Chain of Thought Reasoning in Language Models, published by researchers at Google Research and presented at ICLR 2023.
We'll begin by examining the limitations of Chain-of-Thought prompting that motivated this work, then walk through the intuition behind Self-Consistency, how the decoding algorithm works, and why generating multiple reasoning paths leads to more reliable answers.
Next, we'll analyze the experimental results across arithmetic, common sense, and symbolic reasoning benchmarks, compare Self-Consistency with alternative decoding methods such as beam search and sample-and-rank, and discuss its strengths, limitations, and computational trade-offs.
Finally, we'll examine the paper's long-term impact on language model research and how its central idea influenced later work on test-time reasoning, verification, search-based inference, and modern reasoning-oriented language models.
If you'd like to follow along, you can also read the original paper:
Self-Consistency Improves Chain of Thought Reasoning in Language Models.
And here's a quick infographic of what we'll cover throughout this review.
Table of Contents:
Prerequisites
To get the most out of this review, it helps to be familiar with the evolution of large language models and the reasoning techniques that led to Self-Consistency.
This paper builds directly on the ideas introduced by Chain-of-Thought Prompting, so reading the earlier reviews in this series will provide valuable context.
The previous reviews are especially recommended:
AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)
AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)
AI Paper Review: Language Models are Few-Shot Learners (GPT-3)
AI Paper Review: Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
AI Paper Review: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Among these, the Chain-of-Thought review is the most important prerequisite. It introduced the idea that language models could dramatically improve their reasoning by generating intermediate reasoning steps before producing an answer.
Self-Consistency builds directly on that breakthrough. Instead of trusting a single chain of thought, it explores multiple independent reasoning paths and selects the answer that appears most consistently across them, showing that better reasoning can emerge from a smarter decoding strategy rather than a larger or better-trained model.
It also helps to have:
A general understanding of natural language processing (NLP) and large language models
A basic understanding of Transformer-based autoregressive models
Familiarity with prompting, few-shot learning, in-context learning, and Chain-of-Thought prompting
A high-level understanding of how language models generate text token by token
General machine learning concepts such as training, inference, scaling laws, and model evaluation
Some exposure to reasoning tasks, logic problems, and mathematical word problems
A basic understanding of benchmark datasets and how model performance is evaluated
You don't need a deep background in mathematics or machine learning research to follow this article.
I'll keep the explanations intuitive and practical, focusing on why Self-Consistency became one of the most influential inference-time reasoning techniques in modern AI, how it extended the ideas introduced by Chain-of-Thought prompting, and why a simple change in decoding fundamentally changed how researchers think about reasoning in large language models.
Abstract
The original Chain-of-Thought paper showed that large language models become much better reasoners when they generate intermediate reasoning steps before producing an answer. But it still relied on a simple assumption: the model followed a single reasoning path and trusted its first solution.
This paper asks a natural follow-up question: what if that first reasoning path is wrong?
To answer it, the authors introduce Self-Consistency, a simple decoding strategy inspired by how people often solve difficult problems. Instead of committing to the first chain of thought, the model generates multiple independent reasoning paths and selects the answer that appears most consistently among them.
The model itself remains unchanged. There's no additional training, fine-tuning, or supervision. Only the decoding process is different.
The central insight is that difficult problems rarely have just one valid route to the correct answer. Different reasoning processes may approach a problem in different ways, yet still arrive at the same conclusion. By comparing these independent solutions rather than relying on a single one, the model becomes more robust to reasoning mistakes.
Although the idea is surprisingly simple, its impact is substantial. Self-Consistency significantly improves Chain-of-Thought prompting across arithmetic, common sense, and symbolic reasoning tasks, setting new state-of-the-art results on several popular benchmarks, including GSM8K, SVAMP, AQuA, StrategyQA, and ARC-Challenge.
More importantly, it demonstrated that improving reasoning doesn't always require larger models or additional training. Sometimes, a better way of exploring a model's existing reasoning abilities is enough to produce dramatically better results.
Introduction
When Chain-of-Thought Prompting was introduced in 2022, it changed the conversation around reasoning in large language models. By encouraging models to generate intermediate reasoning steps, researchers discovered that many tasks once considered difficult could suddenly be solved much more effectively.
Yet an important limitation remained: even with Chain-of-Thought, a model still committed to a single reasoning path. If that reasoning contained a mistake, the final answer was likely to be wrong.
The infographic below illustrates the standard Chain-of-Thought reasoning pipeline, showing how a language model follows a single reasoning path using greedy decoding to produce one final answer.
This paper begins with a simple observation: complex problems often have more than one valid route to the correct answer.
People rarely rely on a single line of reasoning when solving difficult problems. Instead, they explore different possibilities and gain confidence when independent approaches lead to the same conclusion. The authors ask whether language models could benefit from the same strategy.
To explore this idea, they introduce Self-Consistency, a decoding strategy that builds directly on Chain-of-Thought prompting. Instead of accepting the first reasoning path the model generates, Self-Consistency samples multiple independent reasoning paths and selects the answer that appears most consistently across them.
The goal is no longer to find a single plausible explanation, but to identify the answer that remains consistent across diverse explanations.
One of the paper's most appealing aspects is its simplicity. Unlike approaches that require additional verifiers, re-ranking models, or extra training, Self-Consistency works entirely at inference time. It requires no new annotations, no fine-tuning, and no auxiliary models. Rather than changing the model itself, it changes only how the model's reasoning is decoded.
The authors evaluate the method across a wide range of arithmetic, common sense, and symbolic reasoning benchmarks using models from UL2 and GPT-3 to LaMDA and PaLM. Across nearly every task, Self-Consistency delivers substantial improvements over standard Chain-of-Thought prompting.
Beyond the impressive benchmark results, the paper introduced a lasting idea: stronger reasoning doesn't always require larger models or more training. Sometimes, the biggest gains come from allowing a model to explore multiple solutions before deciding on the most reliable answer.
Self-Consistency over Diverse Reasoning Paths
The central idea behind this paper begins with a simple observation about human reasoning. When solving difficult problems, people rarely rely on a single line of thought. They often consider multiple possibilities before reaching a conclusion, and although those reasoning processes may differ, they frequently converge on the same answer. The authors argue that language models can benefit from the same principle.
Chain-of-Thought prompting had already shown that generating intermediate reasoning steps could significantly improve performance on complex tasks. But it still relied on greedy decoding, which commits the model to a single reasoning path. If that path contains a mistake, the final answer is likely to be wrong, even if the model could have reached the correct answer through a different line of reasoning.
Self-Consistency replaces this "one path, one answer" strategy with a simple alternative. After receiving a Chain-of-Thought prompt, the model samples multiple reasoning paths instead of selecting only the most likely one. Some paths may contain mistakes, while others may arrive at the correct solution through different reasoning processes.
Rather than evaluating the reasoning itself, the method aggregates the final answers and selects the one that appears most consistently across the generated solutions.
The infographic below compares standard Chain-of-Thought prompting with Self-Consistency, highlighting how replacing a single reasoning path with multiple independent reasoning paths leads to more reliable answers.
The intuition is straightforward. Incorrect reasoning paths tend to make different mistakes and therefore produce different answers. But correct reasoning paths often converge on the same conclusion even when their intermediate steps differ.
By looking for agreement among independent reasoning attempts, the model becomes far less dependent on the success of any single generation.
An elegant aspect of the method is that nothing about the model itself changes. Self-Consistency works entirely at inference time, requiring no additional training, fine-tuning, or auxiliary models. In effect, it behaves like a self-ensemble: instead of combining multiple models, it combines multiple reasoning attempts from the same model to produce a more reliable prediction.
The authors also compare several ways of combining the generated answers. We might expect probability-weighted methods to outperform simpler approaches, but the experiments reveal the opposite. A straightforward majority vote over the final answers performs almost as well as more sophisticated weighting schemes, suggesting that the biggest advantage comes from exploring diverse reasoning paths rather than assigning them complex scores.
This section marks an important shift in how reasoning is viewed. Traditional decoding assumes the most likely reasoning path is also the best one. Self-Consistency shows that, for reasoning tasks, diversity can be just as valuable as confidence. Exploring multiple independent solutions before choosing an answer leads to reasoning that is consistently more robust and reliable.
Experiments
After introducing Self-Consistency, the authors turned to a key question: does this simple decoding strategy actually improve reasoning in practice?
To answer it, they conducted an extensive evaluation across arithmetic, common sense, and symbolic reasoning tasks, testing whether the benefits of Self-Consistency held across different problem types, model architectures, and model sizes.
Rather than relying on a single benchmark, the evaluation spanned a diverse collection of reasoning tasks. Arithmetic datasets measured the ability to solve multi-step math word problems, common sense benchmarks tested reasoning about everyday knowledge, and symbolic tasks evaluated whether models could consistently follow abstract rules.
This broad selection helped determine whether Self-Consistency addresses a general limitation of reasoning rather than improving performance on only a particular dataset.
The experiments also covered a wide range of language models, including UL2, GPT-3, LaMDA, and PaLM, ranging from 20 billion to 540 billion parameters. Evaluating models with different architectures and scales allowed the authors to examine whether the method could generalize beyond a single model family.
To ensure a fair comparison, all experiments remained within the original few-shot Chain-of-Thought prompting framework. The prompts were unchanged, and none of the models were retrained or fine-tuned. As a result, any improvement could be attributed directly to the decoding strategy rather than differences in training or model parameters.
Generating multiple reasoning paths required replacing deterministic greedy decoding with sampling. Instead of always selecting the most likely next token, the model explored several plausible reasoning trajectories.
Although the sampling settings varied slightly across models, the objective was always the same: encourage diverse reasoning paths while maintaining coherent solutions. The authors later investigated how sensitive Self-Consistency is to these sampling choices through a dedicated robustness study.
Overall, the experimental design closely matched the paper's central claim. Rather than introducing larger models, additional supervision, or new training procedures, the authors asked a simpler question: How much better can language models reason if we change only the way they generate and select their answers? The experiments provided a systematic way to answer that question.
The infographic below illustrates the complete Self-Consistency decoding pipeline, showing how a language model generates multiple independent reasoning paths and selects the final answer through majority voting.
Main Results
The central question of this paper is straightforward: does generating multiple reasoning paths and selecting the most consistent answer improve upon the original Chain-of-Thought approach?
The experimental results left little room for doubt. Across nearly every benchmark, model, and reasoning task, Self-Consistency consistently outperformed standard Chain-of-Thought prompting.
The largest improvements appeared in arithmetic reasoning. While Chain-of-Thought had already proven highly effective for solving mathematical word problems, the results showed that much of a model's reasoning ability remained untapped when it relied on a single reasoning path.
By exploring multiple reasoning trajectories before selecting an answer, Self-Consistency achieved substantial gains on challenging benchmarks such as GSM8K, SVAMP, and AQuA, establishing new state-of-the-art results on several of them.
Another interesting pattern emerged as model size increased. Although Self-Consistency benefitted every language model evaluated, the improvements became larger for more capable models.
This suggests that larger models already contain multiple valid reasoning strategies internally, but standard greedy decoding often fails to uncover them. Self-Consistency provides a simple mechanism for making better use of those latent reasoning capabilities.
The improvements were not limited to mathematical reasoning. On common sense reasoning benchmarks such as StrategyQA and ARC-Challenge, as well as symbolic reasoning tasks, Self-Consistency again produced consistent gains over standard Chain-of-Thought prompting.
The fact that the method succeeded across such different problem domains suggests that it addresses a general weakness of greedy decoding rather than exploiting properties of a particular benchmark.
Equally noteworthy is how these improvements were achieved. Unlike many earlier approaches that relied on task-specific fine-tuning, additional verifiers, or auxiliary ranking models, Self-Consistency changed only the decoding process. The language model, prompts, and training remained exactly the same. Yet this simple modification frequently matched or surpassed methods that required additional supervision and specialized training.
Taken together, these results revealed an important insight about reasoning in language models. A model's most likely reasoning path is not necessarily its most reliable one. Allowing several independent reasoning processes to explore the same problem before choosing the answer on which they agree produces reasoning that is consistently more accurate.
More broadly, the paper demonstrates that meaningful improvements in reasoning don't always come from larger models or more training. They can also come from making better use of the reasoning abilities the model already possesses.
Common Sense and Symbolic Reasoning
The strong results on arithmetic reasoning naturally raise a broader question: is Self-Consistency mainly helping with mathematical calculations, or does it improve reasoning more generally?
To answer this, the authors evaluated the method on common sense and symbolic reasoning tasks, two domains that require very different reasoning abilities.
On the common sense benchmarks, Self-Consistency consistently outperformed standard Chain-of-Thought prompting. These tasks require models to reason about everyday situations, make logical inferences, and apply background knowledge rather than perform calculations. The consistent improvements suggested that the method was enhancing the reasoning process itself rather than exploiting properties of mathematical problems.
The symbolic reasoning tasks provided an even tougher test. Instead of relying on world knowledge, models had to follow abstract rules and manipulate symbols correctly. The authors evaluated these tasks in an out-of-distribution setting, where the test problems required longer reasoning chains than those shown in the prompt examples.
Even under these more challenging conditions, Self-Consistency continued to improve performance, particularly for larger language models.
The paper also examined how the number of sampled reasoning paths affected performance. Rather than producing diminishing returns immediately, the results showed a steady improvement as more reasoning paths were generated.
Sampling additional solutions gave the model more opportunities to recover from individual reasoning errors and identify the answer that received the strongest agreement across independent reasoning processes.
To illustrate this behavior, the authors presented several qualitative examples. In one case, greedy decoding confidently produced an incorrect answer after following a flawed reasoning path. When multiple reasoning paths were sampled, however, different solutions independently converged on the correct answer, allowing Self-Consistency to recover from the original mistake.
These examples made the method's intuition tangible: success came not from trusting a single explanation, but from comparing several independent attempts before making a decision.
Together, these experiments reinforced one of the paper's central conclusions. The benefits of Self-Consistency extend well beyond arithmetic reasoning. Whether the task involves everyday knowledge, logical inference, or abstract rule following, allowing multiple reasoning processes to compete before selecting an answer consistently produces more reliable results than relying on a single chain of thought.
Self-Consistency Helps When Chain-of-Thought Hurts Performance
One of the paper's most interesting findings challenged an assumption established by earlier Chain-of-Thought research. Although reasoning traces often improved performance, later studies showed that they were not universally helpful. On some natural language processing tasks, asking a model to explain its reasoning can actually reduce accuracy compared to standard prompting.
This raised an important question: if Chain-of-Thought sometimes hurts performance, can Self-Consistency still help?
To answer this, the authors evaluated Self-Consistency on a collection of question answering and natural language inference benchmarks. Unlike arithmetic reasoning, these tasks often required short, direct responses rather than extended reasoning chains. In such settings, generating a rationale could occasionally distract the model instead of improving its answer.
The results confirmed this behavior. On several benchmarks, standard Chain-of-Thought prompting performed worse than conventional prompting, reinforcing the idea that more reasoning doesn't necessarily lead to better reasoning.
What makes the results particularly compelling is that Self-Consistency largely reversed this trend. Even when individual reasoning paths were imperfect, aggregating multiple independent solutions consistently improved performance. Instead of relying on a single rationale that may have been misleading, the model benefitted from comparing several reasoning attempts before selecting its final answer.
These findings broadened the significance of Self-Consistency. The method isn't limited to mathematical reasoning or tasks that naturally require long chains of thought. It also makes reasoning-based prompting more reliable in situations where generating a rationale can be risky, demonstrating that the value lies not in producing more explanations, but in evaluating multiple independent ones before making a decision.
More broadly, this experiment reinforced one of the paper's central ideas: the effectiveness of Self-Consistency doesn't depend on every reasoning path being correct. It succeeds because correct reasoning paths tend to agree more often than incorrect ones, allowing the model to recover from mistakes that would otherwise determine the final answer.
Comparison to Other Existing Approaches
Once the authors established that Self-Consistency improved reasoning performance, a natural question followed: were these gains simply another manifestation of existing decoding techniques, or did Self-Consistency offer something fundamentally different?
To answer this, the paper compared it with several established approaches for improving generation quality, including sample-and-rank, beam search, and ensemble methods.
Sample-and-Rank
The first comparison was with sample-and-rank, a strategy that generates multiple candidate solutions before selecting the one the model considers most likely.
At first glance, this appears similar to Self-Consistency because both methods generate multiple outputs. The difference lies in how the final answer is chosen. Sample-and-rank still trusts a single reasoning path, whereas Self-Consistency looks for agreement across many independent reasoning paths.
The experiments showed that this distinction mattered: selecting the most consistent answer consistently outperformed selecting the most probable one.
Beam Search
The authors also compared Self-Consistency with beam search, one of the most widely used decoding algorithms in natural language generation.
Beam search explores multiple candidate sequences but favors those with the highest probabilities, often producing reasoning paths that are very similar to one another. Self-Consistency, by contrast, relies on sampling to encourage genuinely different reasoning strategies. This additional diversity proves crucial for reasoning tasks, allowing Self-Consistency to outperform beam search across the evaluated benchmarks.
Ensemble-Based Approaches
The final comparison considers ensemble-based approaches, where diversity is introduced by varying prompt order, using different prompt templates, or combining multiple predictions.
Although these methods provided modest improvements over standard Chain-of-Thought prompting, they fell well short of the gains achieved by Self-Consistency. Remarkably, Self-Consistency accomplished this while using only a single language model and a single prompt.
This comparison highlights one of the paper's most important ideas. Traditional ensembles create diversity by changing prompts or combining multiple models. Self-Consistency discovers diversity within the model itself by allowing it to explore multiple reasoning paths for the same problem. The paper described this as a form of self-ensemble, where different reasoning attempts from a single model collectively determined the final answer.
Taken together, these experiments showed that Self-Consistency is more than another decoding heuristic. Its advantage comes not from generating more outputs or ranking them more carefully, but from exploiting a simple observation: difficult reasoning problems often have multiple valid solution paths, and the answer that consistently emerges across those paths is usually the most reliable one.
Additional Studies
Having established that Self-Consistency improves reasoning performance and outperforms competing decoding methods, the authors devoted the final experimental section to a deeper question: why does the method work so reliably?
Rather than introducing new benchmarks, they investigated how Self-Consistency behaved under different sampling strategies, prompting conditions, and reasoning formats to better understand its robustness.
One of the first findings was that the method remained effective across a variety of sampling strategies. Whether the model used temperature sampling, top-k sampling, or nucleus sampling, the overall improvements remained remarkably consistent.
This suggested that Self-Consistency isn't tied to a particular decoding configuration but instead benefits from the broader idea of exploring multiple reasoning paths before making a decision.
The authors also revisited the relationship between reasoning and model scale. Although models of all sizes benefitted from Self-Consistency, the gains became increasingly pronounced as models grew larger.
This reinforced an important theme throughout the paper: Self-Consistency doesn't create new reasoning abilities. Instead, it helps larger models make better use of reasoning capabilities they already possess.
Another interesting experiment examined imperfect prompts. To simulate realistic conditions, the authors deliberately introduced mistakes into the reasoning demonstrations used for prompting. As expected, greedy decoding became less accurate. Self-Consistency, however, recovered much of the lost performance, showing that it was considerably more robust to flawed reasoning examples than standard Chain-of-Thought prompting.
One of the paper's most intriguing observations concerned the relationship between consistency and correctness. When many sampled reasoning paths converged on the same answer, that answer was much more likely to be correct. Conversely, widespread disagreement among the sampled solutions often signaled uncertainty.
This suggested that Self-Consistency offers more than improved accuracy. It also provides a simple way to estimate the model's confidence by measuring agreement among its own reasoning attempts.
The authors further showed that the method wasn't limited to natural-language reasoning. Replacing verbal reasoning traces with intermediate equations still improved performance, although the gains were smaller because shorter reasoning paths provided less opportunity for diversity.
They also demonstrated that Self-Consistency integrated naturally with Zero-Shot Chain-of-Thought prompting, producing substantial improvements even without manually written reasoning examples.
Taken together, these studies show that Self-Consistency is far more than a decoding trick that works on a handful of benchmarks. Across different sampling strategies, model scales, prompting styles, and reasoning formats, the same pattern continues to emerge: allowing a model to explore multiple reasoning paths before choosing an answer consistently produces reasoning that is both more accurate and more reliable.
Review of Related Work
Self-Consistency didn't emerge in isolation. It has built on several research directions that were already shaping reasoning in language models, combining ideas from prompting, decoding, and consistency into a remarkably simple inference-time strategy.
The most direct influence is Chain-of-Thought prompting, which showed that language models become much better reasoners when they generate intermediate reasoning steps before producing an answer.
Self-Consistency extends that idea by shifting the focus from how a model reasons to how many times it reasons before making a decision. Rather than trusting a single chain of thought, it compares multiple independent reasoning paths and selects the answer on which they agree.
The paper also draws on earlier work in decoding strategies. Techniques such as temperature sampling, top-k sampling, nucleus sampling, and beam search were originally developed to improve text generation by balancing quality and diversity.
Self-Consistency reuses these sampling methods for a different purpose. Instead of generating diverse outputs for creativity, it generates diverse reasoning paths to improve the reliability of a single final answer.
Another closely related area is verification and reranking. Previous approaches often generated multiple candidate solutions and relied on additional verifier models or rerankers (sometimes trained with extra human annotations) to identify the best answer.
Self-Consistency reaches a similar goal without any additional models or supervision. Rather than learning to evaluate reasoning paths, it simply identifies the answer that emerges most consistently across independent reasoning attempts.
Finally, the paper connects to broader research on consistency in language models. Earlier studies examined consistency in conversation, factual knowledge, and generated explanations.
Self-Consistency introduces a different perspective: consistency among multiple reasoning paths. The key insight is that when independent reasoning processes repeatedly converge on the same answer, that agreement itself becomes a strong signal of correctness.
Viewed together, these connections highlight why the paper had such a lasting impact. Self-Consistency didn't require a new model, additional training, or a complex reasoning framework. Instead, it combined existing ideas in a way that fundamentally changed how researchers thought about inference-time reasoning, demonstrating that significant gains could come simply from allowing a model to explore several solutions before choosing the most reliable one.
Discussion
One of the most important ideas in this paper is that better reasoning doesn't necessarily require larger models or more training data. Sometimes, the biggest improvement comes from changing how a model arrives at its final answer.
Rather than trusting the first reasoning path it generates, Self-Consistency allows the model to explore several independent solutions before selecting the answer that receives the strongest agreement. This simple shift changes the role of decoding from choosing the most likely response to identifying the most reliable one.
The experiments suggested that many reasoning failures weren't caused by missing knowledge. Instead, they suggested that a model may already possess the information needed to solve a problem but it fails because it follows an incorrect reasoning path.
By generating multiple reasoning attempts, Self-Consistency gives the model additional opportunities to recover from these mistakes and uncover reasoning capabilities that would otherwise remain hidden.
The paper also highlighted several practical advantages beyond improved benchmark scores. Multiple reasoning paths make it easier to inspect how a model reaches its conclusions, while the level of agreement among those paths provides a useful estimate of confidence.
When independent reasoning processes consistently produce the same answer, that agreement becomes a strong indicator of reliability. Conversely, widespread disagreement can signal uncertainty and identify problems that deserve closer inspection.
Of course, these benefits come with a trade-off. Generating multiple reasoning paths requires additional computation, making inference more expensive than standard Chain-of-Thought prompting. Although the authors showed that much of the improvement could be achieved with a relatively small number of samples, the extra computational cost remains one of the method's primary limitations.
They also noted that incorrect or nonsensical reasoning paths can still be generated. Self-Consistency reduces the impact of these errors, but it can't eliminate them entirely.
More broadly, this paper marked an important shift in how researchers approached reasoning in language models. Earlier work largely focused on improving models through larger architectures, more data, or additional training. Self-Consistency demonstrated that substantial gains could also come from better inference strategies.
That insight has influenced much of the subsequent research on test-time reasoning, search, verification, and the reasoning-oriented language models that followed, making this paper one of the key milestones in the evolution of modern LLM reasoning.
Conclusion
Self-Consistency is a natural continuation of the ideas introduced by Chain-of-Thought prompting.
What appears to be a small change in decoding turns out to have a surprisingly large impact. By replacing a single reasoning path with multiple independent ones and selecting the answer on which they agree, Self-Consistency consistently improves performance across arithmetic, common sense, and symbolic reasoning tasks.
More importantly, it demonstrates that better reasoning doesn't always require larger models or additional training. Sometimes, it simply requires asking the model to think in more than one way.
Looking back, this paper marked an important turning point in the evolution of reasoning in large language models. It shifted the focus from generating the most likely reasoning path to identifying the most reliable answer through agreement among multiple reasoning processes.
That simple idea became the foundation for many later advances in test-time reasoning, search, verification, and the reasoning-oriented language models that followed, securing Self-Consistency's place as one of the most influential papers in modern LLM reasoning.
The infographic below summarizes the key papers that laid the foundation for modern prompting, reasoning, and agentic AI.
Starting with GPT-3's demonstration of in-context learning, it follows the rapid evolution of reasoning techniques, including Zero-Shot Chain-of-Thought, Chain-of-Thought, Self-Consistency, Least-to-Most Prompting, PAL, Program-of-Thoughts, Tree-of-Thoughts, ReAct, and Reflexion.
Collectively, these contributions show how research shifted from simply prompting language models to building systems capable of structured reasoning, planning, tool use, self-reflection, and increasingly autonomous problem solving.
Resources:
Improving Language Understanding by Generative Pre-Training (GPT-1)
Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
Chain of Thought Prompting Elicits Reasoning in Large Language Models
Program Induction by Rationale Generation: Learning to Solve and Explain Algebra Word Problems
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Self-Consistency Improves Chain of Thought Reasoning in Language Models
STaR: Self-Taught Reasoner Bootstrapping Reasoning with Reasoning
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Sparks of Artificial General Intelligence: Early Experiments with GPT-4
Contact Me