# How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives Source: [How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives]() Collection index: [llms.txt]() Data updated: 2025-10-28T13:57:11.917Z Authors: Timour Ichmoukhamedov, James Hinns, David Martens Published: 2024-12-13T15:45:45.000Z ## Abstract A rapidly developing application of LLMs in XAI is to convert quantitative explanations such as SHAP into user-friendly narratives to explain the decisions made by smaller prediction models. Evaluating the narratives without relying on human preference studies or surveys is becoming increasingly important in this field. In this work we propose a framework and explore several automated metrics to evaluate LLM-generated narratives for explanations of tabular classification tasks. We apply our approach to compare several state-of-the-art LLMs across different datasets and prompt types. As a demonstration of their utility, these metrics allow us to identify new challenges related to LLM hallucinations for XAI narratives. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Faithful Narrative Generation on student](): Faithful Narrative Generation; student; RA: 92.5. - [Faithful Narrative Generation on credit](): Faithful Narrative Generation; Credit; RA: 93.8. - [Faithful Narrative Generation on fifa](): Faithful Narrative Generation; fifa; RA Score: 87.5. - [Faithful Narrative Generation on Diabetes](): Faithful Narrative Generation; Diabetes; RA: 0.838. - [Faithful Narrative Generation on stroke](): Faithful Narrative Generation; stroke; RA: 0.925. ## Original sources and implementation [Original paper]() [PDF]()