Per-character Tokenization on MBPP Sanitized
0.32Tokens/Char (MBPP Sanitized)BrahmicTokenizer-131K
Evaluation results
| Method | Links | |
|---|---|---|
| BrahmicTokenizer-131KVocab=131K2026.05 | 0.32 | |
| Tekken/Sarvam-mVocab=131K2026.05 | 0.338 | |
| Sarvam-30BVocab=262K2026.05 | 0.388 | |
| Sarvam-1Vocab=68K2026.05 | 0.424 |