Machine Translation Quality Estimation (Completeness) on WMT DA 2022 (test)
33Completeness Score (AR)STEEREVAL + COMET (wmt22-da)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| STEEREVAL + COMET (wmt22-da)Framework=COMET, Base Model=wmt22-da, Representation=Map2026.01 | 33 | 9 | 9 | 26 | 15 | 16 | -4 | 18 | |
| STEEREVAL + COMET (wmt22-da)Framework=COMET, Base Model=wmt22-da, Representation=Vector2026.01 | 29 | 10 | 13 | 27 | 17 | 17 | 7 | 18 | |
| STEEREVAL + GPTScore (Aya-exp 32b)Framework=GPTScore, Base Model=Aya-exp 32b, Representation=Vector2026.01 | 27 | 7 | 3 | 35 | 18 | 15 | -6 | 14 | |
| STEEREVAL + GPTScore (Aya-exp 32b)Framework=GPTScore, Base Model=Aya-exp 32b, Representation=Map2026.01 | 26 | 6 | 3 | 34 | 18 | 13 | -7 | 13 | |
| STEEREVAL + Direct Prompting (Llama3-8b)Framework=Direct Prompting, Base Model=Llama3-8b, Representation=Map2026.01 | 25 | 3 | -3 | 38 | 15 | 16 | 26 | 15 | |
| STEEREVAL + Direct Prompting (Llama3-8b)Framework=Direct Prompting, Base Model=Llama3-8b, Representation=Vector2026.01 | 23 | 3 | 0 | 39 | 18 | 16 | 26 | 15 | |
| STEEREVAL + GPTScore (Aya-exp 8b)Framework=GPTScore, Base Model=Aya-exp 8b, Representation=Vector2026.01 | 23 | 7 | 1 | 31 | 19 | 11 | -9 | 13 | |
| STEEREVAL + GPTScore (Aya-exp 8b)Framework=GPTScore, Base Model=Aya-exp 8b, Representation=Map2026.01 | 23 | 7 | 1 | 31 | 19 | 10 | -9 | 13 | |
| STEEREVAL + Direct Prompting (Aya-exp 32b)Framework=Direct Prompting, Base Model=Aya-exp 32b, Representation=Vector2026.01 | 21 | 2 | 4 | 35 | 12 | 12 | 7 | 14 | |
| STEEREVAL + Direct Prompting (Aya-exp 32b)Framework=Direct Prompting, Base Model=Aya-exp 32b, Representation=Map2026.01 | 21 | 2 | 4 | 33 | 10 | 12 | 7 | 12 | |
| STEEREVAL + GPTScore (Bloom-7b)Framework=GPTScore, Base Model=Bloom-7b, Representation=Vector2026.01 | 20 | 8 | -4 | 35 | 13 | -3 | -8 | 14 | |
| STEEREVAL + GPTScore (Bloom-7b)Framework=GPTScore, Base Model=Bloom-7b, Representation=Map2026.01 | 19 | 8 | -4 | 31 | 12 | 3 | -8 | 15 | |
| STEEREVAL + GPTScore (Llama3-8b)Framework=GPTScore, Base Model=Llama3-8b, Representation=Vector2026.01 | 15 | 11 | 0 | 40 | 17 | 8 | -6 | 16 | |
| STEEREVAL + GPTScore (Llama3-8b)Framework=GPTScore, Base Model=Llama3-8b, Representation=Map2026.01 | 15 | 11 | 0 | 27 | 17 | 8 | -6 | 15 | |
| STEEREVAL + Direct Prompting (Aya-exp 8b)Framework=Direct Prompting, Base Model=Aya-exp 8b, Representation=Vector2026.01 | 9 | 6 | 12 | 24 | 19 | 15 | 7 | 14 | |
| STEEREVAL + Direct Prompting (Aya-exp 8b)Framework=Direct Prompting, Base Model=Aya-exp 8b, Representation=Map2026.01 | 8 | 6 | 12 | 23 | 14 | 15 | 7 | 14 | |
| STEEREVAL + Direct Prompting (Bloom-7b)Framework=Direct Prompting, Base Model=Bloom-7b, Representation=Vector2026.01 | -12 | -3 | 8 | 27 | -1 | 2 | -4 | 0 | |
| STEEREVAL + Direct Prompting (Bloom-7b)Framework=Direct Prompting, Base Model=Bloom-7b, Representation=Map2026.01 | -14 | 2 | -4 | 10 | -1 | 1 | -6 | -1 |