# Machine Translation Quality Estimation (Coherence) on WMT DA 2022 (test) Source: [Machine Translation Quality Estimation (Coherence) on WMT DA 2022 (test)]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:59:06.812Z Evaluation of Machine Translation Quality Estimation performance, specifically focusing on Coherence, using the WMT DA 2022 test set across multiple language directions. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Machine Translation Quality Estimation (Coherence) | | Dataset (including any stated split) | WMT DA 2022 (test) | | Metric | QE Score (ar) | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [QE Score (ar)](): Higher is better; unit: SCALAR. - [QE Score (es)](): Higher is better; unit: SCALAR. - [QE Score (he)](): Higher is better; unit: SCALAR. - [QE Score (ja)](): Higher is better; unit: SCALAR. - [QE Score (tr)](): Higher is better; unit: SCALAR. - [QE Score (ukr)](): Higher is better; unit: SCALAR. - [QE Score (yo)](): Higher is better; unit: SCALAR. - [QE Score (zh)](): Higher is better; unit: SCALAR. ## Recorded results | Method | QE Score (ar) | Result date | Report paper | | --- | --- | --- | --- | | STEEREVAL + COMET (wmt22-da) | 0.16 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + COMET (wmt22-da) | 0.11 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Llama3-8b) | 0.1 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Llama3-8b) | 0.1 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Bloom-7b) | 0.09 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Bloom-7b) | 0.09 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Aya-exp 32b) | 0.09 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Aya-exp 8b) | 0.08 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Aya-exp 8b) | 0.08 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Aya-exp 32b) | 0.08 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Bloom-7b) | 0.05 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Aya-exp 32b) | 0.04 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Llama3-8b) | 0.04 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + GPTScore (Llama3-8b) | 0.04 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Aya-exp 32b) | 0.03 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Bloom-7b) | 0 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Aya-exp 8b) | 0 | Not recorded | 69730077b2be0ed85a78c981 | | STEEREVAL + Direct Prompting (Aya-exp 8b) | -0.01 | Not recorded | 69730077b2be0ed85a78c981 | ### STEEREVAL + COMET (wmt22-da) QE Score (ar): 0.16. Result ID: 699674d35698665ec4bc367d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"COMET","Base Model":"wmt22-da","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + COMET (wmt22-da) QE Score (ar): 0.11. Result ID: 699674d35698665ec4bc367c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"COMET","Base Model":"wmt22-da","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Llama3-8b) QE Score (ar): 0.1. Result ID: 699674d35698665ec4bc3680. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Llama3-8b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Llama3-8b) QE Score (ar): 0.1. Result ID: 699674d35698665ec4bc3681. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Llama3-8b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Bloom-7b) QE Score (ar): 0.09. Result ID: 699674d35698665ec4bc3686. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Bloom-7b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Bloom-7b) QE Score (ar): 0.09. Result ID: 699674d35698665ec4bc3687. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Bloom-7b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Aya-exp 32b) QE Score (ar): 0.09. Result ID: 699674d35698665ec4bc368c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Aya-exp 32b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Aya-exp 8b) QE Score (ar): 0.08. Result ID: 699674d35698665ec4bc368a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Aya-exp 8b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Aya-exp 8b) QE Score (ar): 0.08. Result ID: 699674d35698665ec4bc368b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Aya-exp 8b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Aya-exp 32b) QE Score (ar): 0.08. Result ID: 699674d35698665ec4bc368d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Aya-exp 32b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Bloom-7b) QE Score (ar): 0.05. Result ID: 699674d35698665ec4bc367f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Bloom-7b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Aya-exp 32b) QE Score (ar): 0.04. Result ID: 699674d35698665ec4bc3684. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Aya-exp 32b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Llama3-8b) QE Score (ar): 0.04. Result ID: 699674d35698665ec4bc3688. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Llama3-8b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + GPTScore (Llama3-8b) QE Score (ar): 0.04. Result ID: 699674d35698665ec4bc3689. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"GPTScore","Base Model":"Llama3-8b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Aya-exp 32b) QE Score (ar): 0.03. Result ID: 699674d35698665ec4bc3685. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Aya-exp 32b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Bloom-7b) QE Score (ar): 0. Result ID: 699674d35698665ec4bc367e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Bloom-7b","Representation":"Vector"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Aya-exp 8b) QE Score (ar): 0. Result ID: 699674d35698665ec4bc3683. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Aya-exp 8b","Representation":"Map"}. This record does not assert independent reproduction. ### STEEREVAL + Direct Prompting (Aya-exp 8b) QE Score (ar): -0.01. Result ID: 699674d35698665ec4bc3682. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Framework":"Direct Prompting","Base Model":"Aya-exp 8b","Representation":"Vector"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.