Decoding LLM's 'Dopamine Neurons': Sparse Subsystems Predict Correctness

Can LLMs Predict Their Own Success? Inside the Sparse Reward Neurons Behind AI Reasoning

Research has repeatedly shown that the hidden states of large language models contain information related to answer correctness, confidence, and hallucination. What has remained unclear is whether such information is distributed across thousands of neurons or concentrated in particular parts of the model. Guowei Xu of Tsinghua University and Mert Yuksekgonul and James Zou of Stanford University push this question further. They argue that reward-related information in LLMs can be concentrated in fewer than 1% of neurons, forming what they call a sparse reward subsystem.

The Model Estimates Whether It Can Solve a Problem Before It Starts Answering

The first component the authors identify is the value neuron. Much like a value function in reinforcement learning, these neurons encode the probability that the model, starting from its current state, will eventually arrive at a correct answer.

What makes the finding striking is that this signal appears even immediately after the question has been entered, before the model generates a single response token. The model has already formed an estimate of its likelihood of success before beginning the solution.

The authors provide an argument for why such information might be concentrated in only a few neurons. Because correctness is ultimately represented by a binary reward, the information needed to predict it can theoretically be compressed into a scalar quantity. They then connect this idea to the superposition hypothesis: features that occur frequently and matter consistently during training may acquire relatively dedicated neural representations.

The experiments follow this prediction. Even after pruning close to 99% of the neurons available to the probe, much of its ability to predict value remains intact.

Remove 1% of the Neurons, and Reasoning Collapses

The most forceful evidence comes not from correlation but from intervention.

The original MATH500 accuracy of Qwen-2.5-7B-SimpleRL-Zoo was 75.2%. When the researchers zeroed out the top 1% of value neurons in individual layers, accuracy fell to 37.0%, 13.6%, 29.4%, and, in one layer, just 1.2%. Across the four layers tested, average accuracy dropped to 20.3%.

When the same proportion of randomly selected neurons was removed, accuracy remained at 74.6%.

The effect was also much larger than removing neurons selected by generic importance criteria such as Magnitude or Wanda. This shifts the paper's argument from “reward information can be decoded from hidden states” toward a stronger claim: the neurons associated with that information participate in the model's reasoning process.

The authors also trained an equivalent probe using a reward-unrelated next-token prediction objective. Its neurons barely overlapped with the value neurons, and ablating them had little effect. The value neurons therefore appear to occupy locations associated specifically with reward-related computation rather than general language generation.

Their locations also show consistency across GSM8K, MATH500, ARC, the coding benchmark MBPP+, and the instruction-following benchmark IFEval. Models fine-tuned differently from the same base model retain more overlap in their critical value neurons than random selection would predict.

Neurons That Register When Reasoning Suddenly Improves or Breaks

The second component is what the authors call the dopamine neuron.

The term does not imply biological identity. It is an analogy to biological dopamine neurons, which are associated with reward prediction errors.

During reasoning, the expected probability of success changes from one step to another. If a new reasoning step suddenly raises the probability of reaching the correct answer, the model experiences a positive temporal-difference, or TD, error. If the step reduces that probability, the TD error becomes negative.

The researchers divide responses into paragraphs and estimate the value of the reasoning state at each boundary by generating multiple continuations. This lets them measure how much each new reasoning step changes the expected final reward.

Figure 3 provides the most intuitive picture of the phenomenon. In a problem that initially appears difficult but is eventually solved, a dopamine neuron produces sharp activation peaks at moments when the model discovers crucial logical steps. In another example, the model begins along a promising path but introduces a logical error midway through the solution. At that exact point, the neuron's activation forms a pronounced trough.

Without reading the generated sentence itself, the activation trajectory indicates something close to: “the reasoning path just improved” or “the reasoning path just deteriorated.”

Similar peak-and-trough patterns appear on the Minerva Math dataset, strengthening the connection between these neuron activations and step-level changes in expected reward.

Reading Confidence Without Asking the Model

The authors then turn these internal signals into practical tools.

Because value neurons contain reward information before response generation begins, they can be used as lightweight confidence estimators. Across the experiments, value neurons achieved an average AUC of 0.67 for predicting whether the model would answer correctly.

A linear classifier using the full hidden state achieved 0.60. Asking the model to verbally report its confidence produced 0.52, while next-token probability produced 0.48.

The result creates an intriguing discrepancy: a small subset of the model's internal activations can predict its chance of success more reliably than the confidence the model expresses in language.

The internal state appears to know something that the verbal response does not report cleanly.

Using the Model's Own Reward Signal to Choose Its Next Thought

The dopamine neurons offer an even more direct application.

At each reasoning stage, the model generates four candidate continuations. The dopamine-based process reward model scores them, and the candidate receiving the highest internal reward signal is selected as the next reasoning step.

On MATH500, greedy decoding achieved 72.2% accuracy. Random selection among candidates also produced 72.2%, while an implicit process reward model reached 75.0%.

Using dopamine-neuron predictions raised accuracy to 77.8%.

The evaluator does not have to be entirely external to the model. A signal emerging inside the model can be read back and used to steer the model's subsequent reasoning.

What the Tables Reveal About the Reward Subsystem

Table 1 compresses the architecture of the paper into two columns.

Value neurons encode the expected value of the current reasoning state and can be used to estimate model confidence. Dopamine neurons encode reward prediction errors between successive states and can therefore function as a process reward model.

The later tables progressively turn this framework into empirical evidence.

Table 2 shows that ablating just 1% of value neurons reduces accuracy by an average of 54.9 percentage points.

Table 3 shows that sparse value neurons outperform several conventional methods for predicting model confidence.

Table 4 shows that dopamine-neuron signals can guide inference and raise MATH500 accuracy to 77.8%.

Together, these tables describe a movement from identification to intervention and finally to application.

From a Vast Hidden State to a Small Reward Circuit

The paper shifts attention away from treating an LLM's hidden state as one enormous undifferentiated representation. Instead, it examines how a small number of neurons behave across time.

Two distinct signals emerge: one tracks whether the current reasoning trajectory is likely to succeed, while another tracks whether the latest reasoning step improved or damaged that trajectory.

The connection between them is also functional. When the researchers ablate value neurons in earlier layers, dopamine neurons in later layers lose their characteristic peaks and troughs. This is why the authors describe the two groups not merely as isolated neurons but as parts of a reward subsystem.

Before an LLM describes its own confidence in words, some of its internal neurons already contain an estimate of success. As reasoning unfolds, another sparse set records changes in that expectation. Most importantly, those signals can be read and fed back into inference to determine which reasoning path the model should follow next.

That is the paper's most consequential image of LLM reasoning: inside the generation process, value and unexpected progress appear as separable neural signals, and those signals can in turn be used to steer the model's next step of reasoning.