# Value-Aware Numerical Representations for Transformer Language Models Source: [Value-Aware Numerical Representations for Transformer Language Models]() Collection index: [llms.txt]() Data updated: 2026-01-16T05:00:11.882Z Authors: Andreea Dutulescu, Stefan Ruseti, Mihai Dascalu Published: 2026-01-14T18:59:14.000Z ## Abstract Transformer-based language models often achieve strong results on mathematical reasoning benchmarks while remaining fragile on basic numerical understanding and arithmetic operations. A central limitation is that numbers are processed as symbolic tokens whose embeddings do not explicitly encode numerical value, leading to systematic errors. We introduce a value-aware numerical representation that augments standard tokenized inputs with a dedicated prefix token whose embedding is explicitly conditioned on the underlying numerical value. This mechanism injects magnitude information directly into the model's input space while remaining compatible with existing tokenizers and decoder-only Transformer architectures. Evaluation on arithmetic tasks shows that the proposed approach outperforms baselines across numerical formats, tasks, and operand lengths. These results indicate that explicitly encoding numerical value is an effective and efficient way to improve fundamental numerical robustness in language models. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Numerical Reasoning on NUPA (aggregated)](): Numerical Reasoning; NUPA (aggregated); Exact Match: 72.4. - [Numerical Generation on NUPA](): Numerical Generation; NUPA; Exact Match: 83.3. ## Original sources and implementation [Original paper]() [PDF]()