Transliteration on Konkani-Bench Devanagari to Kannada
38.71BLEU Scorekonkani-gemma-3-27b-it
Evaluation results
| Method | Links | ||
|---|---|---|---|
| konkani-gemma-3-27b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 38.71 | 72.37 | |
| konkani-Qwen2.5-14B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 37.51 | 72.25 | |
| gemini-flash-latestModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 29.83 | 66.69 | |
| konkani-llama3.1-8bModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 26.88 | 69.69 | |
| DeepSeek-V3.2Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 22.62 | 57.98 | |
| konkani-gemma-3-4b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 22.05 | 61.52 | |
| claude-sonnet-4-6Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 21.11 | 58.32 | |
| konkani-gemma-3-12b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 19.09 | 62.8 | |
| gemma-3-27b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 13.99 | 51.2 | |
| Llama-3.3-70B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 13.82 | 52.29 | |
| kimi-k2-instruct-0905Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 9.08 | 44.46 | |
| konkani-Qwen2.5-1.5B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 8.95 | 39.91 | |
| grok-4-fast-non-reasoningModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 8.28 | 44.47 | |
| gpt-5.2Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 7.79 | 44.76 | |
| gemma-3-12b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 7.1 | 37.2 | |
| command-a-03-2025Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 5.09 | 39.42 | |
| Meta-Llama-3.1-8B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 4.92 | 37.12 | |
| gpt-5-miniModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 1.95 | 33.89 | |
| c4ai-aya-expanse-32bModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 1.3 | 29.7 | |
| Qwen2.5-14B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.66 | 25.66 | |
| Qwen2.5-1.5B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.08 | 7.88 |