Fudan University  ·  ICML 2026 (Spotlight)  ·  arXiv:2510.26277

ICML 2026 · Spotlight

Do LLMs signal when they're right?

Neuron Agreement Decoding — unsupervised answer selection from internal signals

NAD·Neuron Agreement Decoding

Don't read what it answered — watch which neurons agree.

Kang Chen∗ · Yaoning Wang∗ · Kai Xiong · Zhuoka Feng · Wenhe Sun · Haotian Chen · Yixin Cao

Watch the film The idea Read the paper

The one idea

The answer is in the text. The conviction is in the activations.

To pick the best of many sampled reasoning chains without a ground-truth label, decoders usually score the output — token probabilities, entropy, self-evaluations, or a majority vote over answer strings. After post-training those output signals are often poorly calibrated, and a vote needs comparable strings to count. NAD looks one layer down: at which internal neurons each chain switches on while it reasons.

01

Outputs are a projection

Token probabilities and confidence scores are low-dimensional read-outs of a much richer internal state — a thin shadow of the dynamics that produced them.

02

Read the activations instead

For each sampled chain, record the set of neurons that activate above threshold during generation — a fingerprint produced online from the hidden state, never decoded back into text.

03

Select by consensus — read no strings

Score chains by how much their activation sets overlap, and return the one in the densest agreement basin. The answer string is read only once, after the internal-signal decision.

What the activations show

Correct reasoning is sparse, and it agrees with itself.

Across models and benchmarks, two regularities make internal signals usable for selection — correlations, read as evidence, not as proof that a model "knows" it is right.

①

Sparsity

Correct responses activate substantially fewer unique neurons than incorrect ones, throughout generation. Successful chains seem to reach an answer with less trial-and-error; failures over-explore, lighting up many scattered neurons.

②

Agreement

Activations from correct responses show stronger cross-sample agreement — they overlap in neuron space and cluster tightly. Incorrect ones diverge and sit at the margins. Consensus among chains tracks correctness (correlated, not causal).

→

A usable signal

Sparse and self-agreeing means the most-agreed-upon chain is a strong unsupervised pick. NAD turns these two regularities into a single selection rule — no answer strings required.

Three panels of neuron-activation evidence: (a) a t-SNE of activation space where correct reasoning traces cluster tightly and incorrect ones sit at the margins; (b) the distribution of unique activated neurons, with correct responses concentrated low and incorrect ones spread high; (c) cumulative unique neurons over token position, with correct traces staying sparse.
Fig. 1Both regularities, measured. (a) In activation space, correct traces (green) form one tight cluster while incorrect ones (red) scatter to the margins — agreement. (b) Correct responses activate far fewer unique neurons (mean ≈ 11k) than incorrect ones (≈ 24k; Cohen's d ≈ −1.9) — sparsity. (c) The gap appears early and widens over generation.

How NAD works

Take the activated set. Score the overlap. Select the densest consensus.

The NAD pipeline: generate N reasoning paths in parallel; stop early at k tokens; record each path's activated-neuron set; build the pairwise Jaccard similarity matrix; select one path via kNN-agreement, global medoid, or DBSCAN over the neuron-space consensus; continue only the selected path to the answer.
Fig. 2The selector, end to end. N paths are sampled and may be stopped early; each path's activated-neuron set yields a pairwise Jaccard consensus matrix; NAD returns the path in the densest agreement basin (kNN / medoid / DBSCAN) — reading no answer strings.
  1. Ni

    Represent each chain by its activated neurons

    Draw N=64 chains per question. For each chain i, collect the set of neurons that fire above a threshold during generation — a binary activation fingerprint, computed online from the hidden state.

  2. Sij

    Compare chains with Jaccard overlap

    Pairwise set similarity forms an N×N consensus matrix. Jaccard is the shared activated neurons over the total — high when two chains light up the same circuitry.

    Sij = |Ni ∩ Nj||Ni ∪ Nj|
  3. î

    Select the densest agreement basin

    Score each chain by its summed agreement with its k nearest neighbours (kNN-agreement), or its agreement with all others (global medoid). NAD returns the chain at the centre of the densest consensus — no labels, no execution, no answer strings.

    î = arg maxi ∑j∈KNNk(i) Sij

NAD touches token ids only to find generation boundaries, never their contents. It uses no correctness labels, execution results, or answer-token text. Majority voting counts answer strings; NAD aggregates neuron agreement.

Where it lands

On par where voting works. Defined where voting breaks.

Same selector, two regimes: where majority voting is defined, NAD matches it; where it is not, NAD still picks — reading no answer strings.

80.5avg accuracy · 5 benchmarks · Qwen3-4B-Think
−98%fewer tokens · AIME · early stopping
32tokens to first signal

On math & science with verifiable answers, NAD matches majority voting (Cons@64); on open-ended code, where voting is undefined, it beats Avg@64. Everything from internal activations — no answer strings. (NAD-kNN, Qwen3-4B-Think: AIME 85.0 · GPQA 71.7 · HumanEval 97.0 · LCBv5 62.3 · MBPP 86.4.)

Math & science: on par with voting

Reading no answer strings, NAD tracks the majority vote it never sees — AIME24+25 85.0 (vs Cons@64 86.7, Avg@64 74.6) and GPQA 71.7 (above Cons@64's 68.2). The internal-agreement signal recovers what string voting recovers, without the strings.

Open-ended code: voting is undefined, NAD still selects

Code outputs are near-unique strings, so an exact-string majority collapses. The same selector still returns a direct pass@1 and consistently beats Avg@64 — HumanEval 97.0, LiveCodeBench v5 62.3, MBPP 86.4 — exactly where text voting cannot run.

Accuracy and token consumption versus the early-stopping position on a log scale for NAD-kNN: accuracy is already strong at short budgets, while token consumption only climbs steeply at late stopping positions.
Fig. 3Stop early, keep the accuracy. As the stopping position grows (log scale), token cost (red) climbs steeply while accuracy (blue) is already strong at short budgets — the basis for pruning unpromising chains at a fraction of the compute.

A consensus signal, not a verifier

NAD selects the most agreed-upon trajectory, which correlates with correctness — it does not certify it. If a model produces a denser cluster of wrong answers than right ones, NAD can follow it, exactly as textual majority can follow a confident wrong consensus.

Evaluated across 3 reasoning models · 5 benchmarks · 64 samples / question Qwen3-4B-Think, R1-Qwen3-8B, Qwen3-4B-Instruct · AIME24/25, GPQA, HumanEval, LiveCodeBench v5, MBPP · temperature 0.6

Watch

Inside the model — a visual essay

A silent vector essay, drawn live in your browser — no video file. Outputs are a thin shadow of inner dynamics; correct reasoning is sparse and agrees; NAD selects the densest consensus.

Cite

Reference

Kang Chen1,∗, Yaoning Wang1,∗, Kai Xiong2, Zhuoka Feng1, Wenhe Sun1, Haotian Chen1,†, Yixin Cao1,†

1 Fudan University, Institute of Trustworthy Embodied AI    2 Harbin Institute of Technology    ∗ Equal contribution    † Corresponding author

Correspondence: kchen24@m.fudan.edu.cn  ·  yxcao@fudan.edu.cn

Links: Paper (PDF)  ·  Code  ·  arXiv

@misc{chen2025nad,
  title   = {Do LLMs Signal When They're Right? Evidence from Neuron Agreement},
  author  = {Chen, Kang and Wang, Yaoning and Xiong, Kai and Feng, Zhuoka and
             Sun, Wenhe and Chen, Haotian and Cao, Yixin},
  year    = {2025},
  eprint  = {2510.26277},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  note    = {Neuron Agreement Decoding (NAD). ICML 2026 (Spotlight)},
  url     = {https://arxiv.org/abs/2510.26277},
}

Correct = Sparse + Consistent.