Intrinsic Tokenizer Evaluation on Multilingual Indic Corpus (test) benchmark leaderboard · SOTA2 Research

Intrinsic Tokenizer Evaluation on Multilingual Indic Corpus (test)

1.85AS ScoreMUTANT-Indic

Evaluation Results

Method	Links
MUTANT-Indic2025.11		1.85	1.74	2.04	1.47	1.45	1.12	2.17	1.77	1.23	1.21	2.19	1.58	2.3	2.28	1.63	1.62	1.65	1.39	2.59	3.72	1.45	2.12	1.88	1.44
Sutra2025.11		2.12	2.07	3.06	2.12	1.78	1.17	2.68	2.15	1.62	1.48	2.71	2.08	3.1	2.4	2.18	2.01	2.24	1.5	3.76	2.03	2.23	2.58	2.77	1.55
Gemma-32025.11		2.65	1.69	2.84	1.79	1.69	1.39	2.6	2.5	1.47	1.48	3.34	1.91	3.45	2.07	2.03	2.03	4.42	2.83	3.37	5.16	2.03	2.5	2.94	1.44
GPT-OSS2025.11		2.66	2.41	3.17	1.51	1.89	1.33	2.73	2.37	1.72	1.58	3.34	2.01	3.51	2.41	2.61	2.1	6.26	2.71	3.89	13.01	1.76	3.18	3.13	1.51
Sarvam2025.11		4.24	1.91	2.92	2.14	1.85	1.66	3.01	2.11	1.53	1.91	2.53	2.11	3.19	4.6	1.94	2.35	2.43	1.67	3.78	13.07	7.62	2.49	2.63	7.93
LLaMA-42025.11		4.4	2.93	3.34	1.46	2	1.34	2.84	3.37	1.83	1.72	4.23	2.28	4.95	2.73	2.79	2.46	10.51	3.23	4.12	9.04	2.13	5.87	4.53	1.76