Machine Translation Evaluation on WMT Metrics Shared Task 2024
86.4SPAMBR Distill
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MBR DistillIteration (T)=2, Training Protocol=KTO2026.03 | 86.4 | 57.3 | 93.3 | 53.5 | |
| MBR DistillIteration (T)=2, Training Protocol=SFT2026.03 | 86.2 | 56.4 | 93.9 | 56 | |
| MBR DistillIteration (T)=2, Training Protocol=DPO2026.03 | 86 | 57.1 | 93.2 | 53.9 | |
| MBR DistillIteration (T)=3, Training Protocol=SFT2026.03 | 85.7 | 53 | 93.9 | 57.7 | |
| MBR-SCORESIMModel=Llama-3.3-70B-Inst, N=162025.12 | 85.5 | 57.3 | 90.6 | 38.1 | |
| MBR-SCORESIMBackbone=Phi-4, N=642025.12 | 85.2 | 55.9 | 88.1 | 23.8 | |
| Base ModelPrompt style=GEMBA-MQM-style2026.03 | 85.1 | 51.8 | 87.4 | 39.6 | |
| MBR DistillIteration (T)=1, Training Protocol=SFT2026.03 | 85.1 | 57.8 | 92 | 49.7 | |
| MBR-SOFTF1Backbone=Phi-4, N=2562025.12 | 85 | 54.9 | 93.8 | 55.8 | |
| MBR DistillIteration (T)=3, Training Protocol=KTO2026.03 | 85 | 57.1 | 93.9 | 57.5 | |
| MBR-SCORESIMModel=Llama-3.3-70B-Inst, N=642025.12 | 84.9 | 57.8 | 90.2 | 34.7 | |
| MBR DistillIteration (T)=3, Training Protocol=DPO2026.03 | 84.9 | 57.2 | 93.3 | 53.8 | |
| MBR-SOFTF1Backbone=Phi-4, N=642025.12 | 84.8 | 53.5 | 93.8 | 55.9 | |
| MBR-SOFTF1Model=Llama-3.3-70B-Inst, N=2562025.12 | 84.8 | 57.1 | 93.2 | 51.3 | |
| MBR-SOFTF1Backbone=Phi-4, N=162025.12 | 84.7 | 53.5 | 93.6 | 55.2 | |
| MBR-SOFTF1Backbone=Mistral-Large-Inst, N=2562025.12 | 84.6 | 52.8 | 93.4 | 49.4 | |
| MBR-SOFTF1Model=Gemma2-27b-it, N=2562025.12 | 84.6 | 53.2 | 93.8 | 55.3 | |
| MBR-SOFTF1Model=Llama-3.3-70B-Inst, N=162025.12 | 84.5 | 56.5 | 92.9 | 51 | |
| MBR-F1Model=Llama-3.3-70B-Inst, N=2562025.12 | 84.4 | 56.9 | 92.3 | 53.2 | |
| MAPModel=Gemma2-27b-it, N=2562025.12 | 84.4 | 51.9 | 92.6 | 54.5 | |
| MBR-SCORESIMBackbone=Phi-4, N=2562025.12 | 84.3 | 56.3 | 87.5 | 22.2 | |
| MBR-SOFTF1Model=Llama-3.3-70B-Inst, N=642025.12 | 84.3 | 56.9 | 93.1 | 51.4 | |
| MBR-SCORESIMModel=Llama-3.3-70B-Inst, N=2562025.12 | 84.3 | 57.9 | 89.8 | 32.8 | |
| MBR DistillIteration (T)=1, Training Protocol=DPO2026.03 | 84.3 | 56.9 | 92.8 | 52.8 | |
| MBR DistillIteration (T)=1, Training Protocol=KTO2026.03 | 84.2 | 57.3 | 92.5 | 52.9 | |
| MBR-F1Backbone=Phi-4, N=2562025.12 | 84.1 | 53.6 | 91.8 | 57.7 | |
| MAPBackbone=Mistral-Large-Inst, N=2562025.12 | 84.1 | 52.5 | 90.5 | 42.7 | |
| MBR-F1Model=Llama-3.3-70B-Inst, N=642025.12 | 84.1 | 56.8 | 91.3 | 52.9 | |
| MBR-F1Model=Llama-3.3-70B-Inst, N=162025.12 | 83.9 | 56.6 | 91.8 | 52.2 | |
| MBR-SCORESIMBackbone=Phi-4, N=162025.12 | 83.7 | 55.4 | 89.3 | 28.4 | |
| MBR-F1Backbone=Mistral-Large-Inst, N=162025.12 | 83.4 | 53.6 | 88.5 | 45.4 | |
| MBR-F1Backbone=Mistral-Large-Inst, N=2562025.12 | 83.4 | 53.8 | 81.9 | 50.6 | |
| MBR-SOFTF1Model=Gemma2-27b-it, N=642025.12 | 83.3 | 53.3 | 93.8 | 55.4 | |
| MBR-SCORESIMBackbone=Mistral-Large-Inst, N=642025.12 | 83.2 | 54.9 | 85.7 | 10.7 | |
| MAPBackbone=Phi-4, N=2562025.12 | 83.2 | 54.5 | 92.1 | 54.9 | |
| MBR-SOFTF1Model=Gemma2-27b-it, N=162025.12 | 83.2 | 53.1 | 93.5 | 54.5 | |
| MBR-F1Backbone=Phi-4, N=642025.12 | 83.1 | 53 | 92.4 | 57.5 | |
| MBR-SCORESIMBackbone=Mistral-Large-Inst, N=2562025.12 | 83.1 | 55.8 | 85.2 | 10.4 | |
| MAPModel=Llama-3.3-70B-Inst, N=162025.12 | 83.1 | 56.5 | 92.1 | 53.7 | |
| MAPModel=Llama-3.3-70B-Inst, N=642025.12 | 83.1 | 56.8 | 92.2 | 53.9 | |
| MBR-F1Backbone=Mistral-Large-Inst, N=642025.12 | 83 | 53.6 | 87.9 | 49.8 | |
| MBR-SOFTF1Backbone=Mistral-Large-Inst, N=162025.12 | 82.8 | 52.4 | 93 | 48.4 | |
| MBR-SOFTF1Backbone=Mistral-Large-Inst, N=642025.12 | 82.8 | 52.8 | 93.3 | 49.6 | |
| Gold-SFTTraining Protocol=SFT2026.03 | 82.6 | 57.1 | 91.5 | 45.4 | |
| Greedy SearchBackbone=Phi-4, N=12025.12 | 82.5 | 54.8 | 79.2 | 30.7 | |
| MBR-SCORESIMBackbone=Mistral-Large-Inst, N=162025.12 | 82.5 | 53 | 86.5 | 11.9 | |
| MAPBackbone=Mistral-Large-Inst, N=642025.12 | 82.5 | 53 | 91.4 | 47.1 | |
| MAPModel=Llama-3.3-70B-Inst, N=2562025.12 | 82.3 | 56.8 | 91.9 | 53.1 | |
| MBR-F1Backbone=Phi-4, N=162025.12 | 82.2 | 52.8 | 92.5 | 56.3 | |
| MAPBackbone=Phi-4, N=642025.12 | 82.2 | 55.1 | 92.2 | 55 | |
| MBR-F1Model=Gemma2-27b-it, N=642025.12 | 82.1 | 53.7 | 92.2 | 57.5 | |
| Greedy SearchModel=Llama-3.3-70B-Inst, N=12025.12 | 82 | 56.7 | 92.2 | 47.5 | |
| MAPBackbone=Mistral-Large-Inst, N=162025.12 | 81.9 | 54.1 | 91.3 | 46.3 | |
| MAPBackbone=Phi-4, N=162025.12 | 81.7 | 55.3 | 92 | 53.3 | |
| Greedy SearchModel=Gemma2-27b-it, N=12025.12 | 81.6 | 52.4 | 90 | 41.5 | |
| Greedy SearchBackbone=Mistral-Large-Inst, N=12025.12 | 81.5 | 53.8 | 77.4 | 17.1 | |
| MBR-SCORESIMModel=Gemma2-27b-it, N=2562025.12 | 81.5 | 56 | 87.3 | 17 | |
| MBR-SCORESIMModel=Gemma2-27b-it, N=642025.12 | 81.4 | 55.7 | 88 | 19.7 | |
| MBR-F1Model=Gemma2-27b-it, N=2562025.12 | 81.4 | 53.9 | 91.6 | 57.9 | |
| MBR-F1Model=Gemma2-27b-it, N=162025.12 | 81 | 53.8 | 92.4 | 56.1 | |
| MAPModel=Gemma2-27b-it, N=162025.12 | 80.6 | 52.9 | 93.4 | 57.1 | |
| MBR-SCORESIMModel=Gemma2-27b-it, N=162025.12 | 80.2 | 55.2 | 89 | 25.2 | |
| MAPModel=Gemma2-27b-it, N=642025.12 | 79.7 | 52.1 | 93.3 | 56.8 | |
| Gold-KTOTraining Protocol=KTO2026.03 | 68.9 | 56.7 | 91 | 46.9 | |
| Gold-DPOTraining Protocol=DPO2026.03 | 67.3 | 51.8 | 90.9 | 47.2 |