Tokenization Efficiency on FLORES-200 (dev+devtest)
10.43Mean Brahmic FertilityLlama-3.1-8B
Evaluation results
| Method | Links | |
|---|---|---|
| Llama-3.1-8BVocab size=128K2026.05 | 10.43 | |
| Qwen3-8BVocab size=152K2026.05 | 9.36 | |
| IndicBERTv2-SSVocab size=200K2026.05 | 5.23 | |
| DeepSeek-R1Vocab size=129K2026.05 | 5.1 | |
| Krutrim-1Vocab size=70K2026.05 | 5.03 | |
| Tekken/Sarvam-mVocab size=131K2026.05 | 4.87 | |
| GPT-OSS-120BVocab size=200K2026.05 | 3.1 | |
| BrahmicTokenizer-131KVocab size=131K2026.05 | 2.84 | |
| Gemma-3-1BVocab size=262K2026.05 | 2.7 | |
| Sarvam-1Vocab size=68K2026.05 | 2.3 | |
| Sarvam-30BVocab size=262K2026.05 | 2.14 |