# Learning to Deceive with Attention-Based Explanations

Source: [Learning to Deceive with Attention-Based Explanations](<https://www.sota2.com/research/paper/68fa67511e209131cea15642>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:11:14.110Z

Authors: Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton

Published: 2019-09-17T16:10:30.000Z

## Abstract

Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention's reliability as a tool for auditing algorithms in the context of fairness and accountability.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Faithfulness Evaluation on Alpaca 800 samples](<https://www.sota2.com/research/sota/faithfulness-evaluation-on-alpaca-800-samples.md>): Faithfulness Evaluation; Alpaca 800 samples; BLEU: 54.6.

- [Faithfulness Evaluation on tldr\_news (800 samples)](<https://www.sota2.com/research/sota/faithfulness-evaluation-on-tldr_news-800-samples.md>): Faithfulness Evaluation; tldr\_news 800 samples; BLEU: 76.1.

- [Faithfulness Measurement on Alpaca](<https://www.sota2.com/research/sota/faithfulness-measurement-on-alpaca.md>): Faithfulness Measurement; Alpaca; BLEU: 0.547.

- [Faithfulness Measurement on tldr\_news](<https://www.sota2.com/research/sota/faithfulness-measurement-on-tldr_news.md>): Faithfulness Measurement; tldr\_news; BLEU: 76.7.

- [Faithfulness Measurement on MHC](<https://www.sota2.com/research/sota/faithfulness-measurement-on-mhc.md>): Faithfulness Measurement; MHC; BLEU: 68.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1909.07913>)

[PDF](<https://arxiv.org/pdf/1909.07913.pdf>)