# A Quantitative Evaluation of Natural Language Question Interpretation for Question Answering Systems

Source: [A Quantitative Evaluation of Natural Language Question Interpretation for Question Answering Systems](<https://www.sota2.com/research/paper/68fa64d91e209131cea06aaf>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:11:58.696Z

Authors: Takuto Asakura, Jin-Dong Kim, Yasunori Yamamoto, Yuka Tateisi, Toshihisa Takagi

Published: 2018-09-20T05:49:40.000Z

## Abstract

Systematic benchmark evaluation plays an important role in the process of improving technologies for Question Answering (QA) systems. While currently there are a number of existing evaluation methods for natural language (NL) QA systems, most of them consider only the final answers, limiting their utility within a black box style evaluation. Herein, we propose a subdivided evaluation approach to enable finer-grained evaluation of QA systems, and present an evaluation tool which targets the NL question (NLQ) interpretation step, an initial step of a QA pipeline. The results of experiments using two public benchmark datasets suggest that we can get a deeper insight about the performance of a QA system using the proposed approach, which should provide a better guidance for improving the systems, than using black box style approaches.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1809.07485>)

[PDF](<https://arxiv.org/pdf/1809.07485.pdf>)