Machine Translation (English to Kalamang) on MTOB (test)
54.3chrFGemma 4 31B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemma 4 31BModel Size/Architecture=31B, Context length=∼256k (Full book), Thinking mode=without thinking2026.07 | 54.3 | — | |
| Gemma 4 31BModel Size/Architecture=31B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 52.9 | — | |
| Gemma 4 26B-A4BModel Size/Architecture=26B-A4B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 50 | — | |
| Gemma 4 26B-A4BModel Size/Architecture=26B-A4B, Context length=∼256k (Full book), Thinking mode=without thinking2026.07 | 48.9 | — | |
| Gemma 4 12BModel Size/Architecture=12B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 45.1 | — | |
| Gemma 4 12BModel Size/Architecture=12B, Context length=∼256k (Full book), Thinking mode=without thinking2026.07 | 41.9 | — | |
| Gemma 3 27BModel Size/Architecture=27B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 41 | — | |
| Llama-2-70BMethod Setting=Original Baseline (Tanzer et al., 2024)2024.02 | 40.1 | 0 | |
| Gemma 4 E4BModel Size/Architecture=E4B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 37.8 | — | |
| Llama-2-70BMethod Setting=DIPMT++2024.02 | 35.7 | 4.4 | |
| GPT-3.5Method Setting=DIPMT++2024.02 | 35.5 | 2.9 | |
| Llama-2-13BMethod Setting=DIPMT++2024.02 | 31.5 | 3.7 | |
| GPT-3.5Method Setting=Original Baseline (Tanzer et al., 2024)2024.02 | 30.6 | 0 | |
| Llama-2-13BMethod Setting=Original Baseline (Tanzer et al., 2024)2024.02 | 28.8 | 0 | |
| Gemma 4 E2BModel Size/Architecture=E2B, Context length=∼128k (Half book), Thinking mode=without thinking2026.07 | 15.4 | — |