Per-character tokenization on HumanEval
0.295HumanEval Tokens/CharBrahmicTokenizer-131K
Evaluation results
| Method | Links | |
|---|---|---|
| BrahmicTokenizer-131KVocab=131K2026.05 | 0.295 | |
| Tekken/Sarvam-mVocab=131K2026.05 | 0.307 | |
| Sarvam-30BVocab=262K2026.05 | 0.334 | |
| Sarvam-1Vocab=68K2026.05 | 0.378 |