GLEU Without Tuning

About

The GLEU metric was proposed for evaluating grammatical error corrections using n-gram overlap with a set of reference sentences, as opposed to precision/recall of specific annotated errors (Napoles et al., 2015). This paper describes improvements made to the GLEU metric that address problems that arise when using an increasing number of reference sets. Unlike the originally presented metric, the modified metric does not require tuning. We recommend that this version be used instead of the original version.

Courtney Napoles, Keisuke Sakaguchi, Matt Post, Joel Tetreault · 2016

Related benchmarks

Task	Dataset	Result
Correlation with human evaluation	SEEDA E Base	Pearson Correlation (r)0.949	46
GEC Metric Meta-evaluation	SEEDA-E Fluency Setting	Pearson R0.781	17
Correlation with human evaluation	SEEDA-S Base	Pearson Correlation (r)0.886	17
Correlation with human evaluation	SEEDA Fluency (S)	Pearson Correlation (r)0.155	17
Correlation with human evaluation	SEEDA Fluency (E)	Pearson Correlation (r)0.232	17
GEC Metric Meta-evaluation	GMEG-Data	Pearson R0.785	10