# QUACKIE: A NLP Classification Task With Ground Truth Explanations

Source: [QUACKIE: A NLP Classification Task With Ground Truth Explanations](<https://www.sota2.com/research/paper/68fa68fd1e209131cea1f9c5>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:45.965Z

Authors: Yves Rychener, Xavier Renard, Djam\\'e Seddah, Pascal Frossard, Marcin Detyniecki

Published: 2020-12-24T10:43:20.000Z

## Abstract

NLP Interpretability aims to increase trust in model predictions. This makes evaluating interpretability approaches a pressing issue. There are multiple datasets for evaluating NLP Interpretability, but their dependence on human provided ground truths raises questions about their unbiasedness. In this work, we take a different approach and formulate a specific classification task by diverting question-answering datasets. For this custom classification task, the interpretability ground-truth arises directly from the definition of the classification problem. We use this method to propose a benchmark and lay the groundwork for future research in NLP interpretability by evaluating a wide range of current state of the art methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2012.13190>)

[PDF](<https://arxiv.org/pdf/2012.13190.pdf>)
