# Evaluation of Question Answering Systems: Complexity of judging a natural language

Source: [Evaluation of Question Answering Systems: Complexity of judging a natural language](<https://www.sota2.com/research/paper/68fa6d881e209131cea3a6f1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:36:22.217Z

Authors: Amer Farea, Zhen Yang, Kien Duong, Nadeesha Perera, Frank Emmert-Streib

Published: 2022-09-10T12:29:04.000Z

## Abstract

Question answering (QA) systems are among the most important and rapidly developing research topics in natural language processing (NLP). A reason, therefore, is that a QA system allows humans to interact more naturally with a machine, e.g., via a virtual assistant or search engine. In the last decades, many QA systems have been proposed to address the requirements of different question-answering tasks. Furthermore, many error scores have been introduced, e.g., based on n-gram matching, word embeddings, or contextual embeddings to measure the performance of a QA system. This survey attempts to provide a systematic overview of the general framework of QA, QA paradigms, benchmark datasets, and assessment techniques for a quantitative evaluation of QA systems. The latter is particularly important because not only is the construction of a QA system complex but also its evaluation. We hypothesize that a reason, therefore, is that the quantitative formalization of human judgment is an open problem.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2209.12617>)

[PDF](<https://arxiv.org/pdf/2209.12617.pdf>)