# Probing for Arithmetic Errors in Language Models

Source: [Probing for Arithmetic Errors in Language Models](<https://www.sota2.com/research/paper/68fa7a081e209131cea80ab6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:10:24.125Z

Authors: Yucheng Sun, Alessandro Stolfo, Mrinmaya Sachan

Published: 2025-07-16T16:27:50.000Z

## Abstract

We investigate whether internal activations in language models can be used to detect arithmetic errors. Starting with a controlled setting of 3-digit addition, we show that simple probes can accurately decode both the model's predicted output and the correct answer from hidden states, regardless of whether the model's output is correct. Building on this, we train lightweight error detectors that predict model correctness with over 90% accuracy. We then extend our analysis to structured chain-of-thought traces on addition-only GSM8K problems and find that probes trained on simple arithmetic generalize well to this more complex setting, revealing consistent internal representations. Finally, we demonstrate that these probes can guide selective re-prompting of erroneous reasoning steps, improving task accuracy with minimal disruption to correct outputs. Our findings suggest that arithmetic errors can be anticipated from internal activations alone, and that simple probes offer a viable path toward lightweight model self-correction.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Token-level Addition on Addition T3 (val)](<https://www.sota2.com/research/sota/token-level-addition-on-addition-t3-val.md>): Token-level Addition; Addition T3 (val); Token Accuracy: 79.9.

- [Addition on 12-digit addition 3 terms](<https://www.sota2.com/research/sota/addition-on-12-digit-addition-3-terms.md>): Addition; 12-digit addition 3 terms; Token Accuracy: 88.06.

- [Addition on 10-digit addition 3 terms](<https://www.sota2.com/research/sota/addition-on-10-digit-addition-3-terms.md>): Addition; 10-digit addition 3 terms; Token Accuracy: 79.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.12379>)

[PDF](<https://arxiv.org/pdf/2507.12379.pdf>)
