Transliteration on Konkani-Bench Romi to Kannada
67.01BLEU Scoregemini-flash-latest
Evaluation results
| Method | Links | ||
|---|---|---|---|
| gemini-flash-latestModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 67.01 | 87.7 | |
| konkani-Qwen2.5-14B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 43.95 | 74.99 | |
| konkani-gemma-3-27b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 38.38 | 71.66 | |
| konkani-gemma-3-12b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 31.85 | 67.72 | |
| claude-sonnet-4-6Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 30.75 | 67.6 | |
| konkani-llama3.1-8bModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 22.12 | 66.86 | |
| gpt-5.2Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 15.09 | 56.03 | |
| Llama-3.3-70B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 12.22 | 52.19 | |
| konkani-Qwen2.5-1.5B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 11.64 | 36.97 | |
| DeepSeek-V3.2Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 10.97 | 47.77 | |
| kimi-k2-instruct-0905Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 9.92 | 48.24 | |
| konkani-gemma-3-4b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 7.82 | 42.72 | |
| gemma-3-27b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 5.96 | 41.66 | |
| gpt-5-miniModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 5.9 | 41.16 | |
| grok-4-fast-non-reasoningModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 4.16 | 37.78 | |
| gemma-3-12b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 2.94 | 30.48 | |
| command-a-03-2025Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 2.74 | 24.81 | |
| Meta-Llama-3.1-8B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 2.3 | 28.58 | |
| c4ai-aya-expanse-32bModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 1.13 | 29.3 | |
| Qwen2.5-14B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.23 | 17.28 | |
| Qwen2.5-1.5B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.07 | 5.88 |