Transliteration on Konkani-Bench Kannada to Devanagari
56.03BLEU Scorekonkani-Qwen2.5-14B-Instruct
Evaluation results
| Method | Links | ||
|---|---|---|---|
| konkani-Qwen2.5-14B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 56.03 | 79.66 | |
| gemini-flash-latestModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 35.07 | 75.9 | |
| gpt-5.2Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 33.46 | 67.02 | |
| DeepSeek-V3.2Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 31.85 | 65.64 | |
| Llama-3.3-70B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 28.91 | 65.32 | |
| kimi-k2-instruct-0905Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 26.74 | 63.4 | |
| gemma-3-27b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 25.57 | 62.14 | |
| Meta-Llama-3.1-8B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 25.31 | 61.59 | |
| claude-sonnet-4-6Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 23.75 | 57.82 | |
| gpt-5-miniModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 20.41 | 57.81 | |
| konkani-llama3.1-8bModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 19.3 | 49.02 | |
| konkani-gemma-3-27b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 15.91 | 49.03 | |
| grok-4-fast-non-reasoningModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 13.3 | 52.12 | |
| gemma-3-12b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 11.71 | 41.54 | |
| c4ai-aya-expanse-32bModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 11.36 | 48.95 | |
| Qwen2.5-14B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 10.26 | 48.49 | |
| command-a-03-2025Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 5.57 | 34.9 | |
| konkani-gemma-3-12b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 3.51 | 23.55 | |
| konkani-gemma-3-4b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 1.43 | 12.79 | |
| Qwen2.5-1.5B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 1.21 | 10.69 | |
| konkani-Qwen2.5-1.5B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 0.23 | 2.06 |