Tokenization on Wikipedia Cyrillic-rich subsets (test)
2.38Russian (ru)T-pro
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| T-proTokenizer Type=Cyrillic-rich2025.12 | 2.38 | 2.8 | 2.94 | 2.35 | 2.62 | 2.41 | 3.07 | 3.09 | 2.71 | |
| Ruadapt-Qwen32025.12 | 2.43 | 3.29 | 3.54 | 2.5 | 3.07 | 2.7 | 4.6 | 3.97 | 3.26 | |
| GigaChatModel Version=GigaChat-20B-A3B-instruct2025.12 | 2.49 | 3.09 | 3.32 | 2.58 | 2.97 | 2.67 | 2.67 | 3.33 | 2.89 | |
| gpt-oss2025.12 | 2.7 | 2.92 | 3.03 | 2.56 | 2.73 | 2.59 | 3.11 | 3.17 | 2.85 |