# FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation

Source: [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](<https://www.sota2.com/research/paper/68fa6b131e209131cea2bfd7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T05:07:46.501Z

Authors: Moussa Kamal Eddine, Guokan Shang, Antoine J.-P. Tixier, Michalis Vazirgiannis

Published: 2021-10-16T11:59:48.000Z

## Abstract

Fast and reliable evaluation metrics are key to R&D progress. While traditional natural language generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computational resources. In this paper, we propose FrugalScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Experiments with BERTScore and MoverScore on summarization and translation show that FrugalScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running several times faster. On average over all learned metrics, tasks, and variants, FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. We make our trained metrics publicly available, to benefit the entire NLP community and in particular researchers and practitioners with limited resources.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Machine Translation Evaluation on TAC summary-level 2008-2011 (test)](<https://www.sota2.com/research/sota/machine-translation-evaluation-on-tac-summary-level-2008-2011-test.md>): Machine Translation Evaluation; TAC summary-level 2008-2011 (test); Pearson Correlation (Pyramid): 67.3.

- [Machine Translation Evaluation on WMT segment-level 2019 (test)](<https://www.sota2.com/research/sota/machine-translation-evaluation-on-wmt-segment-level-2019-test.md>): Machine Translation Evaluation; WMT segment-level 2019 (test); Pearson R: 43.6.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2110.08559>)

[PDF](<https://arxiv.org/pdf/2110.08559.pdf>)

[Implementation](<https://github.com/moussaKam/FrugalScore>)