# Counterfactual Visual Explanations

Source: [Counterfactual Visual Explanations](<https://www.sota2.com/research/paper/68fa65e11e209131cea0d1df>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:38:30.146Z

Authors: Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, Stefan Lee

Published: 2019-04-16T04:16:11.000Z

## Abstract

In this work, we develop a technique to produce counterfactual visual explanations. Given a 'query' image $I$ for which a vision system predicts class $c$, a counterfactual visual explanation identifies how $I$ could change such that the system would output a different specified class $c'$. To do this, we select a 'distractor' image $I'$ that the system predicts as class $c'$ and identify spatial regions in $I$ and $I'$ such that replacing the identified region in $I$ with the identified region in $I'$ would push the system towards classifying $I$ as $c'$. We apply our approach to multiple image classification datasets generating qualitative results showcasing the interpretability and discriminativeness of our counterfactual explanations. To explore the effectiveness of our explanations in teaching humans, we present machine teaching experiments for the task of fine-grained bird classification. We find that users trained to distinguish bird species fare better when given access to counterfactual explanations in addition to training examples.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Counterfactual Explanation Generation on MNIST (evaluation)](<https://www.sota2.com/research/sota/counterfactual-explanation-generation-on-mnist-evaluation.md>): Counterfactual Explanation Generation; MNIST (evaluation); IM1: 1.44.

- [Counterfactual Explanation Generation on Fashion-MNIST (test)](<https://www.sota2.com/research/sota/counterfactual-explanation-generation-on-fashion-mnist-test.md>): Counterfactual Explanation Generation; Fashion MNIST (test); Image Metric 1: 124.

- [Counterfactual Explanation Generation on MNIST](<https://www.sota2.com/research/sota/counterfactual-explanation-generation-on-mnist.md>): Counterfactual Explanation Generation; MNIST; Inference Time (sec): 0.0291.

- [Counterfactual Explanation Generation on MNIST (Quality Metrics)](<https://www.sota2.com/research/sota/counterfactual-explanation-generation-on-mnist-quality-metrics.md>): Counterfactual Explanation Generation; MNIST; AUPCc': 20.9.

- [Counterfactual Explanation Generation on Fashion-MNIST](<https://www.sota2.com/research/sota/counterfactual-explanation-generation-on-fashion-mnist.md>): Counterfactual Explanation Generation; Fashion MNIST; AUPCc': 27.5.

- [Counterfactual Explanation on MNIST (test)](<https://www.sota2.com/research/sota/counterfactual-explanation-on-mnist-test.md>): Counterfactual Explanation; MNIST (test); Validity Score: 23.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1904.07451>)

[PDF](<https://arxiv.org/pdf/1904.07451.pdf>)