# Machine Translation Meta-evaluation on WMT MQM (En-De, En-Es, Ja-Zh) 24 Source: [Machine Translation Meta-evaluation on WMT MQM (En-De, En-Es, Ja-Zh) 24]() Collection index: [llms.txt]() Data updated: 2026-02-26T06:03:01.348Z Meta-evaluation of machine translation quality using the WMT MQM 2024 dataset, covering English-German, English-Spanish, and Japanese-Chinese language pairs, evaluated using metrics like SPA and accuracy correlation. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Machine Translation Meta-evaluation | | Dataset (including any stated split) | WMT MQM (En-De, En-Es, Ja-Zh) 24 | | Metric | SPA | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 28 of 28 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [SPA](): Higher is better; unit: PERCENT. - [Acc\*Eq](): Higher is better; unit: PERCENT. - [Avg Corr](): Higher is better; unit: PERCENT. ## Recorded results | Method | SPA | Result date | Report paper | | --- | --- | --- | --- | | Remedy-R-14B | 87.9 | Not recorded | 694a22439eebe296e72aaa3d | | XCOMET-XXL | 86.1 | Not recorded | 694a22439eebe296e72aaa3d | | ReMedy9B-24 | 85.9 | Not recorded | 694a22439eebe296e72aaa3d | | Remedy-R-7B | 85.8 | Not recorded | 694a22439eebe296e72aaa3d | | MetricX-24-Hybrid | 85.6 | Not recorded | 694a22439eebe296e72aaa3d | | CometKiwi-XXL | 85.4 | Not recorded | 69784706a2b28e761d87c91d | | Remedy-R-32B | 85.3 | Not recorded | 694a22439eebe296e72aaa3d | | MetricX-24-Hybrid-QE-XXL | 84.9 | Not recorded | 69784706a2b28e761d87c91d | | MetricX-24-Hybrid-QE | 84.9 | Not recorded | 694a22439eebe296e72aaa3d | | GEMBA-ESA | 84.6 | Not recorded | 69784706a2b28e761d87c91d | | GEMBA-ESA (GPT4) | 84.6 | Not recorded | 694a22439eebe296e72aaa3d | | MetricX-24-Hybrid-Large | 84 | Not recorded | 69784706a2b28e761d87c91d | | MetricX-24-Hybrid-QE-XL | 83.4 | Not recorded | 69784706a2b28e761d87c91d | | XCOMET-QE | 83.3 | Not recorded | 69784706a2b28e761d87c91d | | XCOMET-XXL-QE | 83.3 | Not recorded | 694a22439eebe296e72aaa3d | | Skywork-RM | 83.2 | Not recorded | 694a22439eebe296e72aaa3d | | PEAR-XLref,KD | 82.7 | Not recorded | 69784706a2b28e761d87c91d | | PEAR-XLboth,KD | 82.7 | Not recorded | 69784706a2b28e761d87c91d | | PEARref,KD | 82.6 | Not recorded | 69784706a2b28e761d87c91d | | COMET-22 | 82.4 | Not recorded | 69784706a2b28e761d87c91d | | BLEURT-20 | 82.1 | Not recorded | 69784706a2b28e761d87c91d | | PEARboth,KD | 82 | Not recorded | 69784706a2b28e761d87c91d | | PEAR-XLref | 81.6 | Not recorded | 69784706a2b28e761d87c91d | | PEAR-XLboth | 81.6 | Not recorded | 69784706a2b28e761d87c91d | | PEARref | 81.5 | Not recorded | 69784706a2b28e761d87c91d | | PEARboth | 81.2 | Not recorded | 69784706a2b28e761d87c91d | | MetricX-24-Hybrid-QE-Large | 80.6 | Not recorded | 69784706a2b28e761d87c91d | | CometKiwi | 73.3 | Not recorded | 69784706a2b28e761d87c91d | ### Remedy-R-14B SPA: 87.9. Result ID: 6996b64d5698665ec400d575. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":2,"Parameter Scale":"14B"}. This record does not assert independent reproduction. ### XCOMET-XXL SPA: 86.1. Result ID: 6996b64d5698665ec400d577. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":4}. This record does not assert independent reproduction. ### ReMedy9B-24 SPA: 85.9. Result ID: 6996b64d5698665ec400d574. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":1,"Parameter Scale":"9B-24"}. This record does not assert independent reproduction. ### Remedy-R-7B SPA: 85.8. Result ID: 6996b64d5698665ec400d57b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":8,"Parameter Scale":"7B"}. This record does not assert independent reproduction. ### MetricX-24-Hybrid SPA: 85.6. Result ID: 6996b64d5698665ec400d576. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":3}. This record does not assert independent reproduction. ### CometKiwi-XXL SPA: 85.4. Result ID: 69966df65698665ec4b4d4cf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"10.5B","reference\_anchored":false}. This record does not assert independent reproduction. ### Remedy-R-32B SPA: 85.3. Result ID: 6996b64d5698665ec400d57a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":7,"Parameter Scale":"32B"}. This record does not assert independent reproduction. ### MetricX-24-Hybrid-QE-XXL SPA: 84.9. Result ID: 69966df65698665ec4b4d4cb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"13B","reference\_anchored":false}. This record does not assert independent reproduction. ### MetricX-24-Hybrid-QE SPA: 84.9. Result ID: 6996b64d5698665ec400d578. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":5}. This record does not assert independent reproduction. ### GEMBA-ESA SPA: 84.6. Result ID: 69966df65698665ec4b4d4d1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"GPT-4","reference\_anchored":false}. This record does not assert independent reproduction. ### GEMBA-ESA (GPT4) SPA: 84.6. Result ID: 6996b64d5698665ec400d579. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":6,"Backbone":"GPT-4"}. This record does not assert independent reproduction. ### MetricX-24-Hybrid-Large SPA: 84. Result ID: 69966df65698665ec4b4d4d2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"1.2B","reference\_anchored":true}. This record does not assert independent reproduction. ### MetricX-24-Hybrid-QE-XL SPA: 83.4. Result ID: 69966df65698665ec4b4d4cc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"3.7B","reference\_anchored":false}. This record does not assert independent reproduction. ### XCOMET-QE SPA: 83.3. Result ID: 69966df65698665ec4b4d4ce. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"24B","reference\_anchored":false,"ensemble":true}. This record does not assert independent reproduction. ### XCOMET-XXL-QE SPA: 83.3. Result ID: 6996b64d5698665ec400d57c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Rank":9}. This record does not assert independent reproduction. ### Skywork-RM SPA: 83.2. Result ID: 6996b64d5698665ec400d57d. [Reporting paper]() Recorded evaluation configuration: {"Rank":10}. This record does not assert independent reproduction. ### PEAR-XLref,KD SPA: 82.7. Result ID: 69966df65698665ec4b4d4d5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"3.5B","reference\_anchored":true,"knowledge\_distillation":true}. This record does not assert independent reproduction. ### PEAR-XLboth,KD SPA: 82.7. Result ID: 69966df65698665ec4b4d4d6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"3.5B","reference\_anchored":false,"knowledge\_distillation":true}. This record does not assert independent reproduction. ### PEARref,KD SPA: 82.6. Result ID: 69966df65698665ec4b4d4d9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"560M","reference\_anchored":true,"knowledge\_distillation":true}. This record does not assert independent reproduction. ### COMET-22 SPA: 82.4. Result ID: 69966df65698665ec4b4d4d3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"560M","reference\_anchored":true}. This record does not assert independent reproduction. ### BLEURT-20 SPA: 82.1. Result ID: 69966df65698665ec4b4d4d4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"579M","reference\_anchored":true}. This record does not assert independent reproduction. ### PEARboth,KD SPA: 82. Result ID: 69966df65698665ec4b4d4da. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"560M","reference\_anchored":false,"knowledge\_distillation":true}. This record does not assert independent reproduction. ### PEAR-XLref SPA: 81.6. Result ID: 69966df65698665ec4b4d4d7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"3.5B","reference\_anchored":true,"knowledge\_distillation":false}. This record does not assert independent reproduction. ### PEAR-XLboth SPA: 81.6. Result ID: 69966df65698665ec4b4d4d8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"3.5B","reference\_anchored":false,"knowledge\_distillation":false}. This record does not assert independent reproduction. ### PEARref SPA: 81.5. Result ID: 69966df65698665ec4b4d4db. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"560M","reference\_anchored":true,"knowledge\_distillation":false}. This record does not assert independent reproduction. ### PEARboth SPA: 81.2. Result ID: 69966df65698665ec4b4d4dc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"560M","reference\_anchored":false,"knowledge\_distillation":false}. This record does not assert independent reproduction. ### MetricX-24-Hybrid-QE-Large SPA: 80.6. Result ID: 69966df65698665ec4b4d4cd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"1.2B","reference\_anchored":false}. This record does not assert independent reproduction. ### CometKiwi SPA: 73.3. Result ID: 69966df65698665ec4b4d4d0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"theta":"560M","reference\_anchored":false}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.