List your company
ResearchFormerly WizWand

Tokenization Efficiency Analysis on Indic and English corpus

21.09EntropyLLaMA-4
20.393220.574120.75520.9359Nov 5, 2025
Updated 7mo ago

Evaluation results

MethodLinks
LLaMA-42025.1121.0914
GPT-OSS2025.1120.8119
Sarvam2025.1120.7121
Gemma-32025.1120.722
Sutra2025.1120.6223
MUTANT-Indic2025.1120.4228