Machine Translation on FLORES-200 (devtest)
5.67Delta CHRF++SFT 3ep
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SFT 3epModel Parameters=600M, Language=zho_Hant, Language Group=Isol.2026.05 | 5.67 | — | — | — | — | — | — | 0.1 | 0.138 | — | — | |
| GRPOModel Parameters=600M, Language=zho_Hant, Language Group=Isol.2026.05 | 5.07 | — | — | — | — | — | — | 0.092 | 0.115 | — | — | |
| SFT 1epModel Parameters=600M, Language=zho_Hant, Language Group=Isol.2026.05 | 3.74 | — | — | — | — | — | — | 0.069 | 0.102 | — | — | |
| GRPOModel Parameters=600M, Language=bod, Language Group=Aggl.2026.05 | 3.01 | — | — | — | — | — | — | 0.004 | 0.012 | — | — | |
| GRPOModel Parameters=600M, Language=eus, Language Group=Aggl.2026.05 | 2.93 | — | — | — | — | — | — | 0.012 | 0.006 | — | — | |
| SFT 3epModel Parameters=600M, Language=zho_Hans, Language Group=Isol.2026.05 | 2.88 | — | — | — | — | — | — | 0.039 | 0.071 | — | — | |
| SFT 3epModel Parameters=600M, Language=bod, Language Group=Aggl.2026.05 | 2.85 | — | — | — | — | — | — | 0.068 | 0.032 | — | — | |
| SFT 3epModel Parameters=600M, Language=swh, Language Group=Aggl.2026.05 | 2.7 | — | — | — | — | — | — | 0.008 | 0.011 | — | — | |
| SFT 3epModel Parameters=600M, Language=eus, Language Group=Aggl.2026.05 | 2.48 | — | — | — | — | — | — | 0.011 | 0.005 | — | — | |
| SFT 1epModel Parameters=600M, Language=zho_Hans, Language Group=Isol.2026.05 | 2.29 | — | — | — | — | — | — | 0.03 | 0.062 | — | — | |
| GRPOModel Parameters=600M, Language=zho_Hans, Language Group=Isol.2026.05 | 2.28 | — | — | — | — | — | — | 0.034 | 0.04 | — | — | |
| SFT 3epModel Parameters=600M, Language=ben, Language Group=Fus.2026.05 | 2.15 | — | — | — | — | — | — | 0.007 | 0.007 | — | — | |
| GRPOModel Parameters=600M, Language=ben, Language Group=Fus.2026.05 | 1.99 | — | — | — | — | — | — | 0.004 | 0.006 | — | — | |
| GRPOModel Parameters=600M, Language=swh, Language Group=Aggl.2026.05 | 1.88 | — | — | — | — | — | — | 0.004 | 0.008 | — | — | |
| SFT 3epModel Parameters=600M, Language=tur, Language Group=Aggl.2026.05 | 1.8 | — | — | — | — | — | — | 0.005 | 0.005 | — | — | |
| SFT 3epModel Parameters=600M, Language=yor, Language Group=Isol.2026.05 | 1.56 | — | — | — | — | — | — | 0.022 | 0.001 | — | — | |
| SFT 3epModel Parameters=600M, Language=jpn, Language Group=Aggl.2026.05 | 1.46 | — | — | — | — | — | — | 0.006 | 0.007 | — | — | |
| SFT 1epModel Parameters=600M, Language=bod, Language Group=Aggl.2026.05 | 1.43 | — | — | — | — | — | — | 0.035 | 0.017 | — | — | |
| SFT 1epModel Parameters=600M, Language=yor, Language Group=Isol.2026.05 | 1.28 | — | — | — | — | — | — | 0.011 | 0 | — | — | |
| SFT 3epModel Parameters=600M, Language=arb, Language Group=Root-pat.2026.05 | 1.27 | — | — | — | — | — | — | 0.006 | 0.005 | — | — | |
| SFT 1epModel Parameters=600M, Language=swh, Language Group=Aggl.2026.05 | 1.18 | — | — | — | — | — | — | 0.003 | 0.005 | — | — | |
| SFT 1epModel Parameters=600M, Language=eus, Language Group=Aggl.2026.05 | 1.1 | — | — | — | — | — | — | 0.005 | 0.003 | — | — | |
| SFT 3epModel Parameters=600M, Language=ces, Language Group=Fus.2026.05 | 1.08 | — | — | — | — | — | — | 0.005 | 0.003 | — | — | |
| SFT 1epModel Parameters=600M, Language=ben, Language Group=Fus.2026.05 | 1.07 | — | — | — | — | — | — | 0.003 | 0.003 | — | — | |
| GRPOModel Parameters=600M, Language=yor, Language Group=Isol.2026.05 | 0.97 | — | — | — | — | — | — | 0.004 | 0.007 | — | — | |
| SFT 1epModel Parameters=600M, Language=jpn, Language Group=Aggl.2026.05 | 0.96 | — | — | — | — | — | — | 0.012 | 0.009 | — | — | |
| GRPOModel Parameters=600M, Language=tur, Language Group=Aggl.2026.05 | 0.95 | — | — | — | — | — | — | 0.001 | 0.002 | — | — | |
| GRPOModel Parameters=600M, Language=jpn, Language Group=Aggl.2026.05 | 0.86 | — | — | — | — | — | — | 0.018 | 0.011 | — | — | |
| SFT 1epModel Parameters=600M, Language=tur, Language Group=Aggl.2026.05 | 0.68 | — | — | — | — | — | — | 0.002 | 0.002 | — | — | |
| SFT 3epModel Parameters=600M, Language=pol, Language Group=Fus.2026.05 | 0.68 | — | — | — | — | — | — | 0.003 | 0.001 | — | — | |
| GRPOModel Parameters=600M, Language=ces, Language Group=Fus.2026.05 | 0.57 | — | — | — | — | — | — | 0.001 | 0.002 | — | — | |
| GRPOModel Parameters=600M, Language=bel, Language Group=Fus.2026.05 | 0.55 | — | — | — | — | — | — | 0.008 | 0 | — | — | |
| GRPOModel Parameters=600M, Language=pol, Language Group=Fus.2026.05 | 0.55 | — | — | — | — | — | — | 0 | 0.001 | — | — | |
| SFT 3epModel Parameters=600M, Language=bel, Language Group=Fus.2026.05 | 0.43 | — | — | — | — | — | — | 0.007 | 0 | — | — | |
| SFT 1epModel Parameters=600M, Language=arb, Language Group=Root-pat.2026.05 | 0.41 | — | — | — | — | — | — | 0.004 | 0.002 | — | — | |
| SFT 1epModel Parameters=600M, Language=pol, Language Group=Fus.2026.05 | 0.27 | — | — | — | — | — | — | 0.002 | 0.001 | — | — | |
| SFT 1epModel Parameters=600M, Language=ces, Language Group=Fus.2026.05 | 0.17 | — | — | — | — | — | — | 0.002 | 0.001 | — | — | |
| GRPOModel Parameters=600M, Language=arb, Language Group=Root-pat.2026.05 | 0.1 | — | — | — | — | — | — | 0 | 0.001 | — | — | |
| SFT 1epModel Parameters=600M, Language=bel, Language Group=Fus.2026.05 | 0.1 | — | — | — | — | — | — | 0.003 | 0 | — | — | |
| BaseModel Parameters=600M, Language=eus, Language Group=Aggl.2026.05 | — | — | — | — | 52.69 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=swh, Language Group=Aggl.2026.05 | — | — | — | — | 60.33 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=bod, Language Group=Aggl.2026.05 | — | — | — | — | 23.37 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=tur, Language Group=Aggl.2026.05 | — | — | — | — | 56.49 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=jpn, Language Group=Aggl.2026.05 | — | — | — | — | 29.31 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=zho_Hant, Language Group=Isol.2026.05 | — | — | — | — | 16.6 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=zho_Hans, Language Group=Isol.2026.05 | — | — | — | — | 24.59 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=yor, Language Group=Isol.2026.05 | — | — | — | — | 24.28 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=arb, Language Group=Root-pat.2026.05 | — | — | — | — | 54.72 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=bel, Language Group=Fus.2026.05 | — | — | — | — | 41.75 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=ben, Language Group=Fus.2026.05 | — | — | — | — | 49.84 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=ces, Language Group=Fus.2026.05 | — | — | — | — | 55.13 | — | — | — | — | — | — | |
| BaseModel Parameters=600M, Language=pol, Language Group=Fus.2026.05 | — | — | — | — | 48.31 | — | — | — | — | — | — | |
| Final MarianMT Model2026.02 | — | 29.04 | — | — | 58.14 | — | 0.6314 | — | — | — | — | |
| GEMMA-3-12B-PTshots=5, averaging=across all studied translation directions2026.02 | — | 37.51 | 4.06 | 3.63 | 55.37 | 0.84 | — | — | — | — | — | |
| GEMMA-3-1B-PTshots=5, averaging=across all studied translation directions2026.02 | — | 24.55 | 7.29 | 6.59 | 44.12 | 0.71 | — | — | — | — | — | |
| GEMMA-3-270Mshots=5, averaging=across all studied translation directions2026.02 | — | 12.15 | 13.55 | 12.79 | 30.66 | 0.39 | — | — | — | — | — | |
| GEMMA-3-4B-PTshots=5, averaging=across all studied translation directions2026.02 | — | 33.15 | 4.79 | 4.25 | 51.84 | 0.82 | — | — | — | — | — | |
| GPT-OSS-20Bk=–, Expert drop=0%, Params=20.9B, Recovery fine-tuning=None, Calibration direction=N/A2026.05 | — | — | — | — | — | 0.942 | — | — | — | 0.926 | 0.958 | |
| LLAMA-3.2-1Bshots=5, averaging=across all studied translation directions2026.02 | — | 19.34 | 9.27 | 8.48 | 38.58 | 0.59 | — | — | — | — | — | |
| LLAMA-3.2-3Bshots=5, averaging=across all studied translation directions2026.02 | — | 26.74 | 6.55 | 5.78 | 46.02 | 0.73 | — | — | — | — | — | |
| Multiling. En→Xk=16, Expert drop=50.00%, Params=11.3B, Recovery fine-tuning=no retrain, Calibration direction=English→X2026.05 | — | — | — | — | — | 0.93 | — | — | — | 0.909 | 0.951 | |
| Multiling. En→Xk=20, Expert drop=62.50%, Params=9.0B, Recovery fine-tuning=no retrain, Calibration direction=English→X2026.05 | — | — | — | — | — | 0.896 | — | — | — | 0.863 | 0.929 | |
| Multiling. En→Xk=22, Expert drop=68.75%, Params=7.8B, Recovery fine-tuning=no retrain, Calibration direction=English→X2026.05 | — | — | — | — | — | 0.859 | — | — | — | 0.819 | 0.899 | |
| Multiling. En→Xk=22, Expert drop=68.75%, Params=7.8B, Recovery fine-tuning=10k distil., Calibration direction=English→X2026.05 | — | — | — | — | — | 0.904 | — | — | — | 0.874 | 0.934 | |
| Multiling. En→Xk=24, Expert drop=75.00%, Params=6.6B, Recovery fine-tuning=10k distil., Calibration direction=English→X2026.05 | — | — | — | — | — | 0.902 | — | — | — | 0.869 | 0.936 | |
| Multiling. En→Xk=26, Expert drop=81.25%, Params=5.4B, Recovery fine-tuning=10k distil., Calibration direction=English→X2026.05 | — | — | — | — | — | 0.89 | — | — | — | 0.847 | 0.932 | |
| Multiling. En→Xk=28, Expert drop=87.50%, Params=4.2B, Recovery fine-tuning=10k distil., Calibration direction=English→X2026.05 | — | — | — | — | — | 0.871 | — | — | — | 0.812 | 0.93 | |
| NLLB-200k=–, Expert drop=–, Params=3.3B, Recovery fine-tuning=None, Calibration direction=N/A2026.05 | — | — | — | — | — | 0.918 | — | — | — | 0.882 | 0.954 | |
| Previous Best2026.02 | — | 20.76 | — | — | — | — | — | — | — | — | — | |
| QWEN3-0.6B-BASEshots=5, averaging=across all studied translation directions2026.02 | — | 17.47 | 10.83 | 9.87 | 36.36 | 0.55 | — | — | — | — | — | |
| QWEN3-1.7B-BASEshots=5, averaging=across all studied translation directions2026.02 | — | 23.64 | 8.15 | 7.34 | 42.41 | 0.67 | — | — | — | — | — | |
| QWEN3-4B-BASEshots=5, averaging=across all studied translation directions2026.02 | — | 28.16 | 6.71 | 5.98 | 46.74 | 0.73 | — | — | — | — | — | |
| TranslateGemma-4Bk=–, Expert drop=–, Params=4.3B, Recovery fine-tuning=None, Calibration direction=N/A2026.05 | — | — | — | — | — | 0.945 | — | — | — | 0.933 | 0.958 |