Machine Translation on Real-world translation benchmarks En→Ja
34.8Avg ScoreCAT-Translate-3.3B
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CAT-Translate-3.3BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 34.8 | 17.21 | 28.67 | 53.88 | 39.44 | |
| CAT-Translate-7BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 34.31 | 16.29 | 29.62 | 52.94 | 38.37 | |
| CAT-Translate-1.4BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 34.19 | 15.71 | 26.92 | 51.53 | 42.58 | |
| CAT-Translate-0.8BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 30.68 | 14.6 | 26.22 | 50.62 | 32.87 | |
| google/translategemma-12b-itDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 28.68 | 15.92 | 21.79 | 52.53 | 24.47 | |
| Unbabel/Tower-Plus-9BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 27.99 | 10 | 18.8 | 53 | 30.16 | |
| google/translategemma-4b-itDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 26.76 | 14.14 | 20.68 | 51.99 | 20.23 | |
| LiquidAI/LFM2.5-1.2B-JPDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 26.43 | 14.57 | 23.85 | 54.77 | 12.54 | |
| LiquidAI/LFM2-350M-ENJP-MTDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 25 | 8.11 | 22.84 | 47.53 | 21.52 | |
| pfnet/plamo-2-translateDecoding Library=vLLM, Decoding Strategy=Beam Search2026.06 | 24.57 | 17.35 | 24.98 | 32.04 | 23.89 | |
| nvidia/NVIDIA-Nemotron-Nano-9B-v2-JapaneseDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 23.18 | 10.62 | 20.41 | 42.55 | 19.13 | |
| mistralai/Ministral-8B-Instruct-2410Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 20.71 | 12.91 | 16.49 | 41.66 | 11.8 | |
| tencent/HY-MT1.5-1.8BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 20.04 | 6.33 | 18.13 | 37.75 | 17.96 | |
| Rakuten/RakutenAI-2.0-mini-instructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 19.62 | 5.19 | 20.36 | 45.7 | 7.23 | |
| SakanaAI/TinySwallow-1.5B-InstructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 16.49 | 6.3 | 17.58 | 34.07 | 8 | |
| shisa-ai/shisa-v2.1-llama3.2-3bDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 14.28 | 13.18 | 5.54 | 25.97 | 12.41 | |
| llm-jp/llm-jp-3.1-1.8b-instruct4Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 14.11 | 15.38 | 4.91 | 25.47 | 10.65 | |
| shisa-ai/shisa-v2.1-lfm2-1.2bDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 12.14 | 12.11 | 5.54 | 17.6 | 13.3 | |
| google/gemma-2-2b-jpn-itDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 11.32 | 12.37 | 4.48 | 16.24 | 12.21 | |
| microsoft/phi-4Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 10.36 | 3.24 | 6.97 | 14.36 | 16.87 | |
| tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.5Decoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 8.28 | 6.82 | 2.37 | 11.21 | 12.71 | |
| tencent/HY-MT1.5-7BDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 7.67 | 0.82 | 7.8 | 14.3 | 7.74 | |
| Qwen/Qwen2.5-14B-InstructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 6.89 | 4.01 | 3.69 | 13.42 | 6.42 | |
| pfnet/plamo-2-translate2026.06 | 4.429 | 4.486 | 4.514 | 4.714 | 4 | |
| shisa-ai/shisa-v2.1-llama3.2-3bSize=3B2026.06 | 4.409 | 4.114 | 4.494 | 4.571 | 4.455 | |
| tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.5Size=8B2026.06 | 4.323 | 4.286 | 4.265 | 4.286 | 4.455 | |
| shisa-ai/shisa-v2.1-lfm2-1.2bSize=1.2B2026.06 | 4.298 | 4.143 | 4.385 | 4.571 | 4.091 | |
| google/translategemma-12b-itSize=12B2026.06 | 3.924 | 4.257 | 4.153 | 4.286 | 3 | |
| CAT-Translate-1.4BSize=1.4B2026.06 | 3.891 | 4.029 | 4.139 | 4.214 | 3.182 | |
| CAT-Translate-7BSize=7B2026.06 | 3.666 | 3.743 | 4.063 | 3.857 | 3 | |
| google/translategemma-4b-itSize=4B2026.06 | 3.469 | 4.057 | 3.742 | 3.714 | 2.364 | |
| CAT-Translate-0.8BSize=0.8B2026.06 | 3.384 | 3.029 | 3.956 | 3.643 | 2.909 | |
| Qwen/Qwen2.5-14B-InstructSize=14B2026.06 | 3.308 | 3.514 | 3.69 | 4.571 | 1.455 | |
| CAT-Translate-3.3BSize=3.3B2026.06 | 3.29 | 3.2 | 4.066 | 3.714 | 2.182 | |
| microsoft/phi-42026.06 | 3.072 | 3.171 | 2.913 | 3.929 | 2.273 | |
| LiquidAI/LFM2.5-1.2B-JPSize=1.2B2026.06 | 3 | 3.057 | 3.702 | 3.786 | 1.455 | |
| SakanaAI/TinySwallow-1.5B-InstructSize=1.5B2026.06 | 2.782 | 2.886 | 3.138 | 3.286 | 1.818 | |
| google/gemma-2-2b-jpn-itSize=2B2026.06 | 2.721 | 2.943 | 2.342 | 3.143 | 2.455 | |
| tencent/HY-MT1.5-1.8BSize=1.8B2026.06 | 2.588 | 2 | 3.136 | 3.214 | 2 | |
| mistralai/Ministral-8B-Instruct-2410Size=8B2026.06 | 2.462 | 2.486 | 2.868 | 2.857 | 1.636 | |
| LiquidAI/LFM2-350M-ENJP-MTSize=350M2026.06 | 2.461 | 1.314 | 3.29 | 3.786 | 1.455 | |
| meta-llama/Llama-3.2-3B-InstructDecoding Library=transformers, Decoding Strategy=Beam Search, Beam Width=12026.06 | 2.23 | 1.44 | 1.1 | 4.5 | 1.87 | |
| Rakuten/RakutenAI-2.0-mini-instruct2026.06 | 2.069 | 1.257 | 2.805 | 3.214 | 1 | |
| llm-jp/llm-jp-3.1-1.8b-instruct4Size=1.8B2026.06 | 2.008 | 2.086 | 1.686 | 2.714 | 1.545 | |
| tencent/HY-MT1.5-7BSize=7B2026.06 | 1.723 | 1.143 | 1.846 | 2.357 | 1.545 | |
| meta-llama/Llama-3.2-3B-InstructSize=3B2026.06 | 1.166 | 1.171 | 1.351 | 1.143 | 1 |