Tokenization on AI4Bharat Sangraha Total Indic Corpus
6,623Token Count (M)BrahmicTokenizer-131K
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BrahmicTokenizer-131KEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | 6,623 | — | |
| Sarvam-1Evaluation Protocol=27M-document Indic pretraining corpus2026.05 | 5,782 | -12.7 | |
| Sarvam-30BEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | 5,385 | -18.7 | |
| Tekken/SmEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | — | 36.5 |