Tokenization on AI4Bharat Sangraha Assamese
100Token Count (M)Sarvam-1
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Sarvam-1Evaluation Protocol=27M-document Indic pretraining corpus2026.05 | 100 | 49.6 | |
| Sarvam-30BEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | 68 | 1.9 | |
| BrahmicTokenizer-131KEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | 67 | — | |
| Tekken/SmEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | — | 49.4 |