Transliteration on Konkani-Bench Kannada to Romi
55.26BLEUkonkani-Qwen2.5-14B-Instruct
Evaluation results
| Method | Links | ||
|---|---|---|---|
| konkani-Qwen2.5-14B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 55.26 | 81.41 | |
| konkani-gemma-3-27b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 50.11 | 78.92 | |
| gemini-flash-latestModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 47.66 | 79.59 | |
| konkani-gemma-3-12b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 45.05 | 78.49 | |
| konkani-gemma-3-4b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 36.49 | 64.18 | |
| konkani-llama3.1-8bModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 23.43 | 70.13 | |
| gemma-3-27b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 16.56 | 56.03 | |
| gpt-5.2Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 12.58 | 47.76 | |
| DeepSeek-V3.2Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 11.63 | 49.89 | |
| grok-4-fast-non-reasoningModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 10.66 | 47.53 | |
| gemma-3-12b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 10.4 | 47.88 | |
| claude-sonnet-4-6Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 10.08 | 39.51 | |
| Llama-3.3-70B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 10 | 48.06 | |
| kimi-k2-instruct-0905Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 9.96 | 46.56 | |
| gpt-5-miniModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 9.13 | 48.53 | |
| Meta-Llama-3.1-8B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 5.44 | 37.35 | |
| c4ai-aya-expanse-32bModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 4.27 | 37.3 | |
| command-a-03-2025Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 2.79 | 28.41 | |
| Qwen2.5-14B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 2.11 | 27.87 | |
| konkani-Qwen2.5-1.5B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 1.52 | 12.4 | |
| Qwen2.5-1.5B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.27 | 9.55 |