# Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

Source: [Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task](<https://www.sota2.com/research/paper/68fa647f1e209131cea04dfa>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:05:03.439Z

Authors: An Yang, Kai Liu, Jing Liu, Yajuan Lyu, Sujian Li

Published: 2018-06-10T03:50:10.000Z

## Abstract

Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between the candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1806.03578>)

[PDF](<https://arxiv.org/pdf/1806.03578.pdf>)
