Tokenization on Tamil Wikipedia freq >= 3 (Full Evaluation)
1,699,970Total TokensSarvam-1 (68K Indic BPE)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Sarvam-1 (68K Indic BPE)Vocab=68,0962026.03 | 1,699,970 | 3.52 | 23.6 | |
| SentencePiece BPE 16KVocab=16,0002026.03 | 1,375,778 | 2.85 | — | |
| VerChol (morphological)Vocab=12,9912026.03 | 914,619 | 1.89 | 33.5 | |
| VerChol (morphological)Vocab=32,9912026.03 | 896,595 | 1.86 | 34.8 |