# PEDANTS: Cheap but Effective and Interpretable Answer Equivalence Source: [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:29:07.731Z Authors: Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, Jordan Lee Boyd-Graber Published: 2024-02-17T01:56:19.000Z ## Abstract Question answering (QA) can only make progress if we know if an answer is correct, but current answer correctness (AC) metrics struggle with verbose, free-form answers from large language models (LLMs). There are two challenges with current short-form QA evaluations: a lack of diverse styles of evaluation data and an over-reliance on expensive and slow LLMs. LLM-based scorers correlate better with humans, but this expensive task has only been tested on limited QA datasets. We rectify these issues by providing rubrics and datasets for evaluating machine QA adopted from the Trivia community. We also propose an efficient, and interpretable QA evaluation that is more stable than an exact match and neural methods(BERTScore). ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Open-ended Question Answering on TREC-DL-NF (S5)](): Open-ended Question Answering; TREC-DL-NF (S5); Kendall's Tau (K): 48.12. - [Open-ended Question Answering on ANTIQUE (S5)](): Open-ended Question Answering; ANTIQUE (S5); Kendall's Tau (K): 37.85. ## Original sources and implementation [Original paper]() [PDF]()