Machine Translation on Ostraca §3.1
20.02BLEUClaude Opus
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Claude Opus2026.04 | 20.02 | — | 0.46 | |
| GPT-4.1Setting=DEP2026.04 | 18.15 | 0.8859 | 0.37 | |
| GPT-4.1Setting=LEX+SYN2026.04 | 17.99 | 0.9 | 0.53 | |
| GPT-4.1Setting=Baseline2026.04 | 16.25 | 0.8792 | 0.31 | |
| Gemma-3-27bSetting=LEX+SYN2026.04 | 16.18 | 0.8781 | 0.33 | |
| Gemma-3-27bSetting=CoNLLU2026.04 | 15.22 | 0.8622 | 0.21 | |
| GPT-4.1Setting=LEX2026.04 | 14.27 | 0.8973 | 0.42 | |
| Gemma-3-27bSetting=CON2026.04 | 13.32 | 0.855 | 0.13 | |
| Claude Haiku2026.04 | 11.52 | — | 0.35 | |
| Gemma-3-27bSetting=DEP+CON2026.04 | 9.69 | 0.8599 | 0.16 | |
| Gemma-3-27bSetting=LEX2026.04 | 9.44 | 0.8621 | 0.22 | |
| GPT-4.1Setting=CON2026.04 | 9.42 | 0.8888 | 0.31 | |
| GPT-4.1Setting=DEP+CON2026.04 | 9.21 | 0.8862 | 0.32 | |
| CopticTranslator2026.04 | 8.43 | — | 0.3 | |
| Gemma-3-12bSetting=LEX+SYN2026.04 | 7.82 | 0.885 | 0.31 | |
| Gemma-3-12bSetting=DEP2026.04 | 7.5 | 0.847 | 0.11 | |
| Gemma-3-12bSetting=DEP+CON2026.04 | 6.42 | 0.8591 | 0.12 | |
| GPT-4.1Setting=CoNLLU2026.04 | 6.18 | 0.8877 | 0.39 | |
| Gemma-3-12bSetting=Baseline2026.04 | 5.72 | 0.838 | 0.11 | |
| Gemma-3-27bSetting=Baseline2026.04 | 5.61 | 0.849 | 0.13 | |
| Gemma-3-12bSetting=LEX2026.04 | 5.48 | 0.864 | 0.26 | |
| Gemma-3-12bSetting=CoNLLU2026.04 | 4.88 | 0.8536 | 0.18 | |
| Gemma-3-12bSetting=CON2026.04 | 3.84 | 0.852 | 0.1 | |
| Gemma-3-27bSetting=DEP2026.04 | 2.39 | 0.8525 | 0.13 |