Machine Translation on Real-world translation benchmarks Ja→En
41.06Avg ScoreCAT-Translate-7B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| CAT-Translate-7BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 41.06 | 33.75 | 45.29 | 30.65 | 49.86 | 45.74 | |
| CAT-Translate-3.3BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 37.51 | 26.51 | 42.44 | 24.47 | 49.93 | 44.23 | |
| Unbabel/Tower-Plus-9BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 36.84 | 15.43 | 40.54 | 29.13 | 58 | 41.1 | |
| google/translategemma-12b-itDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 35.81 | 31.58 | 34.3 | 23.46 | 48.75 | 40.97 | |
| CAT-Translate-1.4BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 33.26 | 31.28 | 43.84 | 24.08 | 36.55 | 30.57 | |
| CAT-Translate-0.8BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 29.71 | 29.63 | 33.19 | 22.96 | 32.51 | 30.56 | |
| google/translategemma-4b-itDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 29.41 | 28.86 | 25.89 | 21.5 | 42.65 | 28.16 | |
| mistralai/Ministral-8B-Instruct-2410Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 27.52 | 19.23 | 29.21 | 16.25 | 50.23 | 22.69 | |
| pfnet/plamo-2-translateDecoding Library=vLLM, Decoding Strategy=Beam Search2026.06 | 25.92 | 25.55 | 28.63 | 22.9 | 29.02 | 23.48 | |
| LiquidAI/LFM2-350M-ENJP-MTDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 24.91 | 10.94 | 29.56 | 21.48 | 41.4 | 21.17 | |
| LiquidAI/LFM2.5-1.2B-JPDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 24.51 | 19.06 | 29.99 | 22.1 | 43.61 | 7.8 | |
| nvidia/NVIDIA-Nemotron-Nano-9B-v2-JapaneseDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 22.77 | 9.62 | 34.98 | 18.01 | 38.44 | 12.81 | |
| Rakuten/RakutenAI-2.0-mini-instructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 17.24 | 0.11 | 30.62 | 18.21 | 29.34 | 7.9 | |
| google/gemma-2-2b-jpn-itDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 16.98 | 20.04 | 8.08 | 11.27 | 31.49 | 14.01 | |
| llm-jp/llm-jp-3.1-1.8b-instruct4Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 16.26 | 18.82 | 2.44 | 15.67 | 30.65 | 13.72 | |
| SakanaAI/TinySwallow-1.5B-InstructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 14.99 | 4.96 | 18.93 | 15.83 | 26.67 | 8.58 | |
| shisa-ai/shisa-v2.1-llama3.2-3bDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 14.26 | 17.08 | 3.7 | 8.26 | 26.86 | 15.42 | |
| shisa-ai/shisa-v2.1-lfm2-1.2bDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 14.02 | 20.93 | 4.95 | 7.68 | 26.72 | 9.8 | |
| microsoft/phi-4Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 13.48 | 6.1 | 18.66 | 2.81 | 24.86 | 14.98 | |
| tencent/HY-MT1.5-7BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 13.46 | 4.99 | 12.32 | 5.72 | 29.53 | 14.76 | |
| tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.5Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 12.42 | 24.25 | 2.3 | 3.69 | 14.11 | 17.74 | |
| meta-llama/Llama-3.2-3B-InstructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 9.9 | 18.6 | 0.41 | 2.72 | 16.62 | 11.17 | |
| Qwen/Qwen2.5-14B-InstructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 9.88 | 10.81 | 4.7 | 4.27 | 11.18 | 18.46 | |
| tencent/HY-MT1.5-1.8BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 8.95 | 5.5 | 4.59 | 4 | 15.67 | 14.98 |