Machine Translation on Ostraca §3.1 benchmark leaderboard · SOTA2 Research

Machine Translation on Ostraca §3.1

20.02BLEUClaude Opus

Evaluation Results

Method	Links
Claude Opus2026.04		20.02	—	0.46
GPT-4.12026.04		18.15	0.8859	0.37
GPT-4.12026.04		17.99	0.9	0.53
GPT-4.12026.04		16.25	0.8792	0.31
Gemma-3-27b2026.04		16.18	0.8781	0.33
Gemma-3-27b2026.04		15.22	0.8622	0.21
GPT-4.12026.04		14.27	0.8973	0.42
Gemma-3-27b2026.04		13.32	0.855	0.13
Claude Haiku2026.04		11.52	—	0.35
Gemma-3-27b2026.04		9.69	0.8599	0.16
Gemma-3-27b2026.04		9.44	0.8621	0.22
GPT-4.12026.04		9.42	0.8888	0.31
GPT-4.12026.04		9.21	0.8862	0.32
CopticTranslator2026.04		8.43	—	0.3
Gemma-3-12b2026.04		7.82	0.885	0.31
Gemma-3-12b2026.04		7.5	0.847	0.11
Gemma-3-12b2026.04		6.42	0.8591	0.12
GPT-4.12026.04		6.18	0.8877	0.39
Gemma-3-12b2026.04		5.72	0.838	0.11
Gemma-3-27b2026.04		5.61	0.849	0.13
Gemma-3-12b2026.04		5.48	0.864	0.26
Gemma-3-12b2026.04		4.88	0.8536	0.18
Gemma-3-12b2026.04		3.84	0.852	0.1
Gemma-3-27b2026.04		2.39	0.8525	0.13