# NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction

Source: [NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction](<https://www.sota2.com/research/paper/6916b79c0052cac1dc65e163>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-11-15T05:02:48.629Z

Authors: Peter R{\\o}ysland Aarnes, Vinay Setty

Published: 2025-11-13T05:09:52.000Z

## Abstract

Large language models show strong performance on knowledge intensive tasks such as fact-checking and question answering, yet they often struggle with numerical reasoning. We present a systematic evaluation of state-of-the-art models for veracity prediction on numerical claims and evidence pairs using controlled perturbations, including label-flipping probes, to test robustness. Our results indicate that even leading proprietary systems experience accuracy drops of up to 62\\% under certain perturbations. No model proves to be robust across all conditions. We further find that increasing context length generally reduces accuracy, but when extended context is enriched with perturbed demonstrations, most models substantially recover. These findings highlight critical limitations in numerical fact-checking and suggest that robustness remains an open challenge for current language models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2511.09971>)

[PDF](<https://arxiv.org/pdf/2511.09971.pdf>)
