Tokenization Efficiency on IN22-Gen (test)
10.97Mean Brahmic FertilityLlama-3.1-8B
Evaluation results
| Method | Links | |
|---|---|---|
| Llama-3.1-8BVocab size=128K2026.05 | 10.97 | |
| Qwen3-8BVocab size=152K2026.05 | 9.84 | |
| IndicBERTv2-SSVocab size=200K2026.05 | 5.5 | |
| DeepSeek-R1Vocab size=129K2026.05 | 5.43 | |
| Krutrim-1Vocab size=70K2026.05 | 5.27 | |
| Tekken/Sarvam-mVocab size=131K2026.05 | 5.21 | |
| GPT-OSS-120BVocab size=200K2026.05 | 3.37 | |
| BrahmicTokenizer-131KVocab size=131K2026.05 | 3.08 | |
| Gemma-3-1BVocab size=262K2026.05 | 2.92 | |
| Sarvam-1Vocab size=68K2026.05 | 2.57 | |
| Sarvam-30BVocab size=262K2026.05 | 2.35 |