# Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations

Source: [Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations](<https://www.sota2.com/research/paper/68fa7a0d1e209131cea80d2f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-09T05:00:06.005Z

Authors: Pedro Ferreira, Wilker Aziz, Ivan Titov

Published: 2025-04-07T17:49:23.000Z

## Abstract

Chain-of-thought explanations are widely used to inspect the decision process of large language models (LLMs) and to evaluate the trustworthiness of model outputs, making them important for effective collaboration between LLMs and humans. We demonstrate that preference optimization - a key step in the alignment phase - can inadvertently reduce the faithfulness of these explanations. This occurs because the reward model (RM), which guides alignment, is tasked with optimizing both the expected quality of the response and the appropriateness of the explanations (e.g., minimizing bias or adhering to safety standards), creating potential conflicts. The RM lacks a mechanism to assess the consistency between the model's internal decision process and the generated explanation. Consequently, the LLM may engage in "reward hacking" by producing a final response that scores highly while giving an explanation tailored to maximize reward rather than accurately reflecting its reasoning. To address this issue, we propose enriching the RM's input with a causal attribution of the prediction, allowing the RM to detect discrepancies between the generated self-explanation and the model's decision process. In controlled settings, we show that this approach reduces the tendency of the LLM to generate misleading explanations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Chain-of-Thought Faithfulness Evaluation on Math Book](<https://www.sota2.com/research/sota/chain-of-thought-faithfulness-evaluation-on-math-book.md>): Chain-of-Thought Faithfulness Evaluation; Math Book; Unfaithful Explanation Rate: 22.8.

- [Chain-of-Thought Faithfulness Evaluation on BiasQA](<https://www.sota2.com/research/sota/chain-of-thought-faithfulness-evaluation-on-biasqa.md>): Chain-of-Thought Faithfulness Evaluation; BiasQA; Unfaithful Explanation Rate: 2.4.

- [Explanation Faithfulness Evaluation on Math Book (test)](<https://www.sota2.com/research/sota/explanation-faithfulness-evaluation-on-math-book-test.md>): Explanation faithfulness evaluation; Math Book (test); Greedy Score: 31.4.

- [Explanation Faithfulness Evaluation on BiasQA (test)](<https://www.sota2.com/research/sota/explanation-faithfulness-evaluation-on-biasqa-test.md>): Explanation faithfulness evaluation; BiasQA (test); Greedy: 18.2.

- [Code Evaluation on CodeMMLU software principles](<https://www.sota2.com/research/sota/code-evaluation-on-codemmlu-software-principles.md>): Code evaluation; CodeMMLU software principles; Accuracy: 23.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2504.05294>)

[PDF](<https://arxiv.org/pdf/2504.05294.pdf>)
