Machine Translation Quality Estimation (Coherence) on WMT DA 2022 (test)
0.16QE Score (ar)STEEREVAL + COMET (wmt22-da)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| STEEREVAL + COMET (wmt22-da)Framework=COMET, Base Model=wmt22-da, Representation=Map2026.01 | 0.16 | 0.23 | 0.03 | 0.14 | 0.05 | 0.12 | -0.05 | 0.14 | |
| STEEREVAL + COMET (wmt22-da)Framework=COMET, Base Model=wmt22-da, Representation=Vector2026.01 | 0.11 | 0.23 | 0.04 | 0.14 | 0.11 | 0.14 | -0.02 | 0.15 | |
| STEEREVAL + Direct Prompting (Llama3-8b)Framework=Direct Prompting, Base Model=Llama3-8b, Representation=Vector2026.01 | 0.1 | 0.2 | -0.01 | 0.25 | 0.09 | 0.2 | 0.13 | 0.21 | |
| STEEREVAL + Direct Prompting (Llama3-8b)Framework=Direct Prompting, Base Model=Llama3-8b, Representation=Map2026.01 | 0.1 | 0.16 | 0.01 | 0.25 | 0.03 | 0.2 | 0.12 | 0.19 | |
| STEEREVAL + GPTScore (Bloom-7b)Framework=GPTScore, Base Model=Bloom-7b, Representation=Vector2026.01 | 0.09 | 0.23 | 0 | 0.18 | 0.1 | 0.03 | -0.04 | 0.2 | |
| STEEREVAL + GPTScore (Bloom-7b)Framework=GPTScore, Base Model=Bloom-7b, Representation=Map2026.01 | 0.09 | 0.23 | -0.03 | 0.18 | 0.1 | 0.02 | -0.04 | 0.23 | |
| STEEREVAL + GPTScore (Aya-exp 32b)Framework=GPTScore, Base Model=Aya-exp 32b, Representation=Vector2026.01 | 0.09 | 0.19 | -0.05 | 0.15 | 0.08 | 0.14 | -0.04 | 0.2 | |
| STEEREVAL + GPTScore (Aya-exp 8b)Framework=GPTScore, Base Model=Aya-exp 8b, Representation=Vector2026.01 | 0.08 | 0.2 | -0.02 | 0.18 | 0.09 | 0.15 | -0.07 | 0.2 | |
| STEEREVAL + GPTScore (Aya-exp 8b)Framework=GPTScore, Base Model=Aya-exp 8b, Representation=Map2026.01 | 0.08 | 0.2 | -0.03 | 0.18 | 0.09 | 0.15 | -0.07 | 0.2 | |
| STEEREVAL + GPTScore (Aya-exp 32b)Framework=GPTScore, Base Model=Aya-exp 32b, Representation=Map2026.01 | 0.08 | 0.18 | -0.05 | 0.13 | 0.08 | 0.14 | -0.04 | 0.18 | |
| STEEREVAL + Direct Prompting (Bloom-7b)Framework=Direct Prompting, Base Model=Bloom-7b, Representation=Map2026.01 | 0.05 | 0.07 | 0 | 0.2 | 0.05 | -0.05 | -0.03 | 0.08 | |
| STEEREVAL + Direct Prompting (Aya-exp 32b)Framework=Direct Prompting, Base Model=Aya-exp 32b, Representation=Vector2026.01 | 0.04 | 0.17 | 0.03 | 0.2 | -0.02 | 0.19 | 0.1 | 0.18 | |
| STEEREVAL + GPTScore (Llama3-8b)Framework=GPTScore, Base Model=Llama3-8b, Representation=Vector2026.01 | 0.04 | 0.23 | -0.04 | 0.18 | 0.07 | 0.14 | -0.06 | 0.23 | |
| STEEREVAL + GPTScore (Llama3-8b)Framework=GPTScore, Base Model=Llama3-8b, Representation=Map2026.01 | 0.04 | 0.23 | -0.03 | 0.17 | 0.07 | 0.13 | -0.06 | 0.23 | |
| STEEREVAL + Direct Prompting (Aya-exp 32b)Framework=Direct Prompting, Base Model=Aya-exp 32b, Representation=Map2026.01 | 0.03 | 0.16 | 0.04 | 0.18 | -0.03 | 0.18 | 0.11 | 0.15 | |
| STEEREVAL + Direct Prompting (Bloom-7b)Framework=Direct Prompting, Base Model=Bloom-7b, Representation=Vector2026.01 | 0 | 0.01 | 0.05 | 0.19 | 0.03 | 0.01 | 0.02 | 0.09 | |
| STEEREVAL + Direct Prompting (Aya-exp 8b)Framework=Direct Prompting, Base Model=Aya-exp 8b, Representation=Map2026.01 | 0 | 0.11 | 0.07 | 0.25 | 0.04 | 0.21 | 0.05 | 0.16 | |
| STEEREVAL + Direct Prompting (Aya-exp 8b)Framework=Direct Prompting, Base Model=Aya-exp 8b, Representation=Vector2026.01 | -0.01 | 0.12 | 0.08 | 0.28 | 0.04 | 0.22 | 0.05 | 0.14 |