Tokenization on AI4Bharat Sangraha Marathi
529,000,000Token CountBrahmicTokenizer-131K
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BrahmicTokenizer-131KEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | 529,000,000 | — | |
| Sarvam-30BEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | 414,000,000 | -21.7 | |
| Sarvam-1Evaluation Protocol=27M-document Indic pretraining corpus2026.05 | 408,000,000 | -22.9 | |
| Tekken/SmEvaluation Protocol=27M-document Indic pretraining corpus2026.05 | — | 24.3 |