Morphosyntactic Transformation on ConLang Benchmark Average over 9 languages
40.94TER (Translation Error Rate)gemini-2.5-pro
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| gemini-2.5-proICL=Yes2025.10 | 40.94 | 12.64 | 16.71 | 14.68 | 89.3 | |
| gemini-2.5-flashICL=Yes2025.10 | 48.35 | 18.02 | 24.14 | 21.08 | 89.99 | |
| claude-3-5-sonnetICL=Yes2025.10 | 55.22 | 26.29 | 24.18 | 25.23 | 89.13 | |
| gpt-5ICL=Yes2025.10 | 57.57 | 35.7 | 33.9 | 34.8 | 79.9 | |
| gemini-2.5-flashICL=No2025.10 | 74.86 | 43.07 | 47.64 | 45.36 | 73.11 | |
| claude-3-5-sonnetICL=No2025.10 | 77.78 | 33.37 | 41.73 | 37.55 | 76.17 | |
| gpt-5ICL=No2025.10 | 78.53 | 45.58 | 46.26 | 45.92 | 61.76 | |
| gpt-5-miniICL=No2025.10 | 80.8 | 62.7 | 47.8 | 55.25 | 68.28 | |
| gemini-2.5-proICL=No2025.10 | 83 | 60.8 | 47.57 | 54.18 | 74.38 | |
| gpt-5-miniICL=Yes2025.10 | 84.76 | 62.79 | 46.01 | 54.4 | 69.13 | |
| gpt-4o-miniICL=No2025.10 | 87.58 | 71.64 | 66.67 | 69.16 | 52.79 | |
| gpt-4o-miniICL=Yes2025.10 | 88.83 | 69.93 | 66.36 | 68.14 | 53.49 |