# Benchmarking XAI Explanations with Human-Aligned Evaluations

Source: [Benchmarking XAI Explanations with Human-Aligned Evaluations](<https://www.sota2.com/research/paper/68fa7ae41e209131cea85b7a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:20.825Z

Authors: R\\'emi Kazmierczak, Steve Azzolin, Elo\\"ise Berthier, Anna Hedstr\\"om, Patricia Delhomme, David Filliat, Nicolas Bousquet, Goran Frehse, Massimiliano Mancini, Baptiste Caramiaux, Andrea Passerini, Gianni Franchi

Published: 2024-11-04T15:18:20.000Z

## Abstract

We introduce PASTA (Perceptual Assessment System for explanaTion of Artificial Intelligence), a novel human-centric framework for evaluating eXplainable AI (XAI) techniques in computer vision. Our first contribution is the creation of the PASTA-dataset, the first large-scale benchmark that spans a diverse set of models and both saliency-based and concept-based explanation methods. This dataset enables robust, comparative analysis of XAI techniques based on human judgment. Our second contribution is an automated, data-driven benchmark that predicts human preferences using the PASTA-dataset. This scoring called PASTA-score method offers scalable, reliable, and consistent evaluation aligned with human perception. Additionally, our benchmark allows for comparisons between explanations across different modalities, an aspect previously unaddressed. We then propose to apply our scoring method to probe the interpretability of existing models and to build more human interpretable XAI methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Explanation Quality Assessment on Soccer xG Explanations (User Study)](<https://www.sota2.com/research/sota/explanation-quality-assessment-on-soccer-xg-explanations-user-study.md>): Explanation Quality Assessment; Soccer xG Explanations (User Study); AUROC: 0.53.

- [Explanation Quality Rejection on sleepstage](<https://www.sota2.com/research/sota/explanation-quality-rejection-on-sleepstage.md>): Explanation quality rejection; sleepstage; AUROC: 62.

- [Explanation Quality Rejection on Compas (AUROC)](<https://www.sota2.com/research/sota/explanation-quality-rejection-on-compas-auroc.md>): Explanation quality rejection; COMPAS; AUROC: 0.75.

- [Explanation quality rejection on iknl](<https://www.sota2.com/research/sota/explanation-quality-rejection-on-iknl.md>): Explanation quality rejection; iknl; AUROC: 60.

- [Explanation quality rejection on celeba](<https://www.sota2.com/research/sota/explanation-quality-rejection-on-celeba.md>): Explanation quality rejection; CelebA; AUROC: 0.65.

- [Explanation quality rejection on adult](<https://www.sota2.com/research/sota/explanation-quality-rejection-on-adult.md>): Explanation quality rejection; Adult; AUROC: 0.58.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.02470>)

[PDF](<https://arxiv.org/pdf/2411.02470.pdf>)
