# SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References

Source: [SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References](<https://www.sota2.com/research/paper/68fa70af1e209131cea4c90b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:01.294Z

Authors: Matteo Gabburo, Siddhant Garg, Rik Koncel Kedziorski, Alessandro Moschitti

Published: 2023-09-21T16:51:30.000Z

## Abstract

Evaluation of QA systems is very challenging and expensive, with the most reliable approach being human annotations of correctness of answers for questions. Recent works (AVA, BEM) have shown that transformer LM encoder based similarity metrics transfer well for QA evaluation, but they are limited by the usage of a single correct reference answer. We propose a new evaluation metric: SQuArE (Sentence-level QUestion AnsweRing Evaluation), using multiple reference answers (combining multiple correct and incorrect references) for sentence-form QA. We evaluate SQuArE on both sentence-level extractive (Answer Selection) and generative (GenQA) QA systems, across multiple academic and industrial datasets, and show that it outperforms previous baselines and obtains the highest correlation with human annotations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2309.12250>)

[PDF](<https://arxiv.org/pdf/2309.12250.pdf>)