# Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

Source: [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](<https://www.sota2.com/research/paper/68fa688e1e209131cea1cf4f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:40:59.400Z

Authors: Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, Ankur P. Parikh

Published: 2020-10-08T23:16:26.000Z

## Abstract

The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem. This paper describes our contribution to the WMT 2020 Metrics Shared Task, the main benchmark for automatic evaluation of translation. We make several submissions based on BLEURT, a previously published metric based on transfer learning. We extend the metric beyond English and evaluate it on 14 language pairs for which fine-tuning data is available, as well as 4 "zero-shot" language pairs, for which we have no labelled examples. Additionally, we focus on English to German and demonstrate how to combine BLEURT's predictions with those of YiSi and use alternative reference translations to enhance the performance. Empirical results show that the models achieve competitive results on the WMT Metrics 2019 Shared Task, indicating their promise for the 2020 edition.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [MBR decoding on WMT En-De 2024 (test)](<https://www.sota2.com/research/sota/mbr-decoding-on-wmt-en-de-2024-test.md>): MBR decoding; WMT En-De 2024 (test); XCOMET-XL: 84.2.

- [MBR decoding on WMT En-Ja 2024 (test)](<https://www.sota2.com/research/sota/mbr-decoding-on-wmt-en-ja-2024-test.md>): MBR decoding; WMT En-Ja 2024 (test); XCOMET-XL: 0.796.

- [Machine Translation Meta-evaluation on WMT MQM (En-De, En-Es, Ja-Zh) 24](<https://www.sota2.com/research/sota/machine-translation-meta-evaluation-on-wmt-mqm-en-de-en-es-ja-zh-24.md>): Machine Translation Meta-evaluation; WMT MQM (En-De, En-Es, Ja-Zh) 24; SPA: 82.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.04297>)

[PDF](<https://arxiv.org/pdf/2010.04297.pdf>)