Tokenization Intrinsic Evaluation on mC4 unbalanced 60-lang (test)
0.0212Compression RatePA-BPE
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| PA-BPEVocabulary size=128k2025.08 | 0.0212 | 2.64 | 58.9 | 3.707 | 52 | 8.3 | 28.8 | 46.6 | |
| PA-BPE (window)Vocabulary size=128k2025.08 | 0.0218 | 3.21 | 70.1 | 3.647 | 51 | 10.1 | 31.4 | 46.9 | |
| PA-BPE (hybrid)Vocabulary size=128k2025.08 | 0.0219 | 2.79 | 60.4 | 3.624 | 52 | 10 | 29.3 | 45 | |
| BPEVocabulary size=128k2025.08 | 0.0222 | 3.41 | 72.8 | 3.564 | 51 | 14.4 | 33.5 | 48.1 | |
| PA-BPE (window+hybrid)Vocabulary size=128k2025.08 | 0.0222 | 3.38 | 72.1 | 3.576 | 50 | 11.6 | 32.3 | 46.6 |