Transliteration on Konkani-Bench Devanagari to Romi
50.52BLEUkonkani-Qwen2.5-14B-Instruct
Evaluation results
| Method | Links | ||
|---|---|---|---|
| konkani-Qwen2.5-14B-InstructModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 50.52 | 80.04 | |
| konkani-gemma-3-12b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 48.56 | 78.07 | |
| konkani-gemma-3-27b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 45.81 | 76.41 | |
| gemini-flash-latestModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 43.83 | 77.34 | |
| konkani-gemma-3-4b-itModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 42.19 | 70.41 | |
| konkani-llama3.1-8bModel Category=Konkani Finetuned Models, Sample Size=200 samples2026.03 | 30.36 | 74.13 | |
| gpt-5.2Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 13.35 | 46.7 | |
| DeepSeek-V3.2Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 12.24 | 49.47 | |
| gemma-3-27b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 11.83 | 49.53 | |
| claude-sonnet-4-6Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 10.48 | 43.42 | |
| kimi-k2-instruct-0905Model Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 8.51 | 42.41 | |
| grok-4-fast-non-reasoningModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 7.87 | 41.92 | |
| Llama-3.3-70B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 7.08 | 42.6 | |
| gpt-5-miniModel Category=Proprietary & Large API Models, Sample Size=200 samples2026.03 | 6.57 | 43.54 | |
| gemma-3-12b-itModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 4.84 | 38.78 | |
| Meta-Llama-3.1-8B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 4.72 | 37.35 | |
| c4ai-aya-expanse-32bModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 2.92 | 34.85 | |
| Qwen2.5-14B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 1.3 | 21.4 | |
| konkani-Qwen2.5-1.5B-InstructModel Category=Konkani Finetuned Models, Sample Size=160 samples2026.03 | 0.97 | 13.1 | |
| Qwen2.5-1.5B-InstructModel Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.3 | 8.46 | |
| command-a-03-2025Model Category=Open-Weights (General), Sample Size=200 samples2026.03 | 0.19 | 10.34 |