Translation on Konkani-Bench 200 samples (test)
86.76BLEUgemini-flash-latest
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| gemini-flash-latestCategory=Proprietary & Large API Models2026.03 | 86.76 | 92.13 | 91.14 | |
| gemini-3-pro-previewCategory=Proprietary & Large API Models2026.03 | 71.5 | 83.73 | 88.56 | |
| gpt-5.2Category=Proprietary & Large API Models2026.03 | 57.37 | 74.04 | 85.93 | |
| claude-sonnet-4-6Category=Proprietary & Large API Models2026.03 | 57.16 | 74.65 | 84.43 | |
| konkani-Qwen2.5-14B-InstructCategory=Konkani Finetuned Models2026.03 | 50.26 | 68.98 | 83.58 | |
| konkani-gemma-3-27b-itCategory=Konkani Finetuned Models2026.03 | 48.76 | 67.67 | 82.91 | |
| kimi-k2-instruct-0905Category=Proprietary & Large API Models2026.03 | 46.4 | 64.05 | 80.41 | |
| DeepSeek-V3.2Category=Open-Weights (General)2026.03 | 46.05 | 63.11 | 80.91 | |
| konkani-gemma-3-12b-itCategory=Konkani Finetuned Models2026.03 | 41.96 | 62.48 | 80.48 | |
| konkani-llama3.1-8bCategory=Konkani Finetuned Models2026.03 | 37.6 | 62.23 | 79.18 | |
| gpt-5-miniCategory=Proprietary & Large API Models2026.03 | 34.87 | 54.76 | 75.53 | |
| konkani-gemma-3-4b-itCategory=Konkani Finetuned Models2026.03 | 34.1 | 58.02 | 79.08 | |
| gemma-3-27b-itCategory=Open-Weights (General)2026.03 | 32.64 | 52.59 | 73.87 | |
| grok-4-fast-non-reasoningCategory=Proprietary & Large API Models2026.03 | 29.38 | 51.22 | 71.72 | |
| Llama-3.3-70B-InstructCategory=Open-Weights (General)2026.03 | 24.74 | 45.11 | 69.97 | |
| gemma-3-12b-itCategory=Open-Weights (General)2026.03 | 18.18 | 39.21 | 66.3 | |
| Meta-Llama-3.1-8B-InstructCategory=Open-Weights (General)2026.03 | 6.85 | 28.2 | 57.1 | |
| c4ai-aya-expanse-32bCategory=Open-Weights (General)2026.03 | 6.27 | 28.24 | 58.13 | |
| Qwen2.5-14B-InstructCategory=Open-Weights (General)2026.03 | 5.59 | 25.77 | 56.9 | |
| konkani-Qwen2.5-1.5B-InstructCategory=Konkani Finetuned Models2026.03 | 2.68 | 14.61 | 36.94 | |
| Qwen2.5-1.5B-InstructCategory=Open-Weights (General)2026.03 | 2.11 | 19.56 | 49.88 | |
| command-a-03-2025Category=Open-Weights (General)2026.03 | 1.92 | 18.66 | 59.18 |