Machine Translation (Kalamang to English) on MTOB (test)
48.6chrFGemma 4 31B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemma 4 31BModel Size/Architecture=31B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 48.6 | — | |
| Gemma 4 31BModel Size/Architecture=31B, Context length=∼256k (Full book), Thinking mode=without thinking2026.07 | 46.2 | — | |
| Gemma 4 26B-A4BModel Size/Architecture=26B-A4B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 45 | — | |
| Gemma 4 26B-A4BModel Size/Architecture=26B-A4B, Context length=∼256k (Full book), Thinking mode=without thinking2026.07 | 42.7 | — | |
| Gemma 4 12BModel Size/Architecture=12B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 37.3 | — | |
| GPT-3.5Method Setting=DIPMT++2024.02 | 37 | 11.4 | |
| Llama-2-70BMethod Setting=DIPMT++2024.02 | 36.3 | 12.3 | |
| Llama-2-13BMethod Setting=DIPMT++2024.02 | 35.2 | 11.3 | |
| Llama-2-70BMethod Setting=Original Baseline (Tanzer et al., 2024)2024.02 | 34.9 | 9.4 | |
| Gemma 4 E4BModel Size/Architecture=E4B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 34.6 | — | |
| Gemma 4 12BModel Size/Architecture=12B, Context length=∼256k (Full book), Thinking mode=without thinking2026.07 | 32.9 | — | |
| Gemma 3 27BModel Size/Architecture=27B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 31.2 | — | |
| GPT-3.5Method Setting=Original Baseline (Tanzer et al., 2024)2024.02 | 31.1 | 6.6 | |
| Llama-2-13BMethod Setting=Original Baseline (Tanzer et al., 2024)2024.02 | 29.8 | 4.8 | |
| Gemma 4 E2BModel Size/Architecture=E2B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 28.2 | — |