Machine Translation on Real-world translation benchmarks Ja→En (M-Prometheus)
4.589M-Prometheus (Macro Avg.)shisa-ai/shisa-v2.1-llama3.2-3b
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| shisa-ai/shisa-v2.1-llama3.2-3bSize=3B2026.06 | 4.589 | 4.733 | 4.559 | 4.44 | 4.842 | 4.9 | 4.923 | |
| pfnet/plamo-2-translate2026.06 | 4.525 | 4.602 | 4.676 | 4.44 | 4.657 | 4.7 | 4.538 | |
| tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.5Size=8B2026.06 | 4.501 | 4.644 | 4.676 | 4.64 | 4.581 | 4.4 | 4.923 | |
| shisa-ai/shisa-v2.1-lfm2-1.2bSize=1.2B2026.06 | 4.37 | 4.427 | 4.265 | 4.72 | 4.845 | 5 | 3.308 | |
| google/translategemma-12b-itSize=12B2026.06 | 4.264 | 4.536 | 4.441 | 4.78 | 4.803 | 4.5 | 4.154 | |
| CAT-Translate-7BSize=7B2026.06 | 4.084 | 4.418 | 3.971 | 4.88 | 4.777 | 5 | 3.462 | |
| CAT-Translate-1.4BSize=1.4B2026.06 | 4.034 | 4.148 | 3.294 | 4.7 | 4.748 | 5 | 3 | |
| google/translategemma-4b-itSize=4B2026.06 | 3.929 | 4.297 | 4 | 4.6 | 4.662 | 4.3 | 3.923 | |
| CAT-Translate-3.3BSize=3.3B2026.06 | 3.73 | 4.082 | 3.735 | 4.84 | 4.803 | 4.8 | 2.231 | |
| CAT-Translate-0.8BSize=0.8B2026.06 | 3.657 | 3.876 | 2.735 | 4.72 | 4.661 | 4.8 | 2.462 | |
| Qwen/Qwen2.5-14B-InstructSize=14B2026.06 | 3.545 | 3.736 | 4.412 | 3.94 | 3.38 | 3.1 | 3.846 | |
| microsoft/phi-42026.06 | 3.342 | 3.558 | 4.206 | 3.2 | 2.792 | 4.9 | 2.692 | |
| google/gemma-2-2b-jpn-itSize=2B2026.06 | 3.241 | 3.657 | 3.882 | 3.16 | 4.075 | 4.4 | 2.769 | |
| LiquidAI/LFM2.5-1.2B-JPSize=1.2B2026.06 | 3.172 | 3.31 | 2 | 4.44 | 4.355 | 4.6 | 1.154 | |
| mistralai/Ministral-8B-Instruct-2410Size=8B2026.06 | 3.147 | 3.695 | 3.794 | 4 | 3.698 | 4.6 | 2.385 | |
| SakanaAI/TinySwallow-1.5B-InstructSize=1.5B2026.06 | 3.036 | 3.239 | 2.647 | 3.78 | 3.82 | 4.1 | 1.846 | |
| LiquidAI/LFM2-350M-ENJP-MTSize=350M2026.06 | 2.816 | 3.1 | 1.353 | 4.06 | 4.003 | 4.7 | 1.385 | |
| llm-jp/llm-jp-3.1-1.8b-instruct4Size=1.8B2026.06 | 2.355 | 2.633 | 1.941 | 2.08 | 3.66 | 4.1 | 1.385 | |
| Rakuten/RakutenAI-2.0-mini-instruct2026.06 | 2.222 | 2.344 | 1.147 | 2.94 | 3.001 | 3.4 | 1.231 | |
| tencent/HY-MT1.5-1.8BSize=1.8B2026.06 | 2.109 | 1.726 | 2.353 | 1.3 | 1.567 | 2.1 | 1.308 | |
| tencent/HY-MT1.5-7BSize=7B2026.06 | 2.074 | 2.355 | 1.588 | 2.46 | 1.825 | 3.9 | 2 | |
| meta-llama/Llama-3.2-3B-InstructSize=3B2026.06 | 1.837 | 2.373 | 3 | 1.3 | 2.402 | 3.7 | 1.462 |