Translation on Konkani-Bench 200 samples (test) benchmark leaderboard · SOTA2 Research

Translation on Konkani-Bench 200 samples (test)

86.76BLEUgemini-flash-latest

Evaluation Results

Method	Links
gemini-flash-latest2026.03		86.76	92.13	91.14
gemini-3-pro-preview2026.03		71.5	83.73	88.56
gpt-5.22026.03		57.37	74.04	85.93
claude-sonnet-4-62026.03		57.16	74.65	84.43
konkani-Qwen2.5-14B-Instruct2026.03		50.26	68.98	83.58
konkani-gemma-3-27b-it2026.03		48.76	67.67	82.91
kimi-k2-instruct-09052026.03		46.4	64.05	80.41
DeepSeek-V3.22026.03		46.05	63.11	80.91
konkani-gemma-3-12b-it2026.03		41.96	62.48	80.48
konkani-llama3.1-8b2026.03		37.6	62.23	79.18
gpt-5-mini2026.03		34.87	54.76	75.53
konkani-gemma-3-4b-it2026.03		34.1	58.02	79.08
gemma-3-27b-it2026.03		32.64	52.59	73.87
grok-4-fast-non-reasoning2026.03		29.38	51.22	71.72
Llama-3.3-70B-Instruct2026.03		24.74	45.11	69.97
gemma-3-12b-it2026.03		18.18	39.21	66.3
Meta-Llama-3.1-8B-Instruct2026.03		6.85	28.2	57.1
c4ai-aya-expanse-32b2026.03		6.27	28.24	58.13
Qwen2.5-14B-Instruct2026.03		5.59	25.77	56.9
konkani-Qwen2.5-1.5B-Instruct2026.03		2.68	14.61	36.94
Qwen2.5-1.5B-Instruct2026.03		2.11	19.56	49.88
command-a-03-20252026.03		1.92	18.66	59.18