# PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation

Source: [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](<https://www.sota2.com/research/paper/69784706a2b28e761d87c91d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-29T05:00:12.873Z

Authors: Lorenzo Proietti, Roman Grundkiewicz, Matt Post

Published: 2026-01-25T21:52:30.000Z

## Abstract

We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised quality estimation (QE) metric family that reframes reference-free machine translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for minimum Bayes risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [MBR decoding on WMT En-De 2024 (test)](<https://www.sota2.com/research/sota/mbr-decoding-on-wmt-en-de-2024-test.md>): MBR decoding; WMT En-De 2024 (test); XCOMET-XL: 85.5.

- [MBR decoding on WMT En-Ja 2024 (test)](<https://www.sota2.com/research/sota/mbr-decoding-on-wmt-en-ja-2024-test.md>): MBR decoding; WMT En-Ja 2024 (test); XCOMET-XL: 0.81.

- [Machine Translation Meta-evaluation on WMT MQM (En-De, En-Es, Ja-Zh) 24](<https://www.sota2.com/research/sota/machine-translation-meta-evaluation-on-wmt-mqm-en-de-en-es-ja-zh-24.md>): Machine Translation Meta-evaluation; WMT MQM (En-De, En-Es, Ja-Zh) 24; SPA: 82.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.18006>)

[PDF](<https://arxiv.org/pdf/2601.18006.pdf>)