# Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics

Source: [Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics](<https://www.sota2.com/research/paper/68fa6c561e209131cea335fc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:08:48.414Z

Authors: Daniel Deutsch, Dan Roth

Published: 2022-04-21T15:43:45.000Z

## Abstract

Question answering-based summarization evaluation metrics must automatically determine whether the QA model's prediction is correct or not, a task known as answer verification. In this work, we benchmark the lexical answer verification methods which have been used by current QA-based metrics as well as two more sophisticated text comparison methods, BERTScore and LERC. We find that LERC out-performs the other methods in some settings while remaining statistically indistinguishable from lexical overlap in others. However, our experiments reveal that improved verification performance does not necessarily translate to overall QA-based metric quality: In some scenarios, using a worse verification method -- or using none at all -- has comparable performance to using the best verification method, a result that we attribute to properties of the datasets.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2204.10206>)

[PDF](<https://arxiv.org/pdf/2204.10206.pdf>)