Per-character tokenization on GSM8K (train)
0.381Tokens per Character Ratio (GSM8K train)Sarvam-1
Evaluation results
| Method | Links | |
|---|---|---|
| Sarvam-1Vocab=68K2026.05 | 0.381 | |
| Tekken/Sarvam-mVocab=131K2026.05 | 0.351 | |
| Sarvam-30BVocab=262K2026.05 | 0.351 | |
| BrahmicTokenizer-131KVocab=131K2026.05 | 0.301 |