Tokenization Efficiency on 1.5-million sentence evaluation corpus Hindi
18,394,075Other TokensOpenAI (o200k)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| OpenAI (o200k)2026.03 | 18,394,075 | — | 1.37 | |
| Llama 4 Scout2026.03 | 19,566,121 | — | 1.46 | |
| DeepSeek V32026.03 | 31,682,218 | — | 2.36 | |
| SGPE2026.03 | — | 13,433,554 | — |