Tokenizer Evaluation on mC4 unbalanced 30-lang
0.0296Compression RatePA-BPE (window+hybrid-127k)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| PA-BPE (window+hybrid-127k)Vocabulary Size=127k2025.08 | 0.0296 | 0.1231 | 49.3 | 3.698 | 0.46 | 0.032 | 0.53 | 0.596 | |
| BPEVocabulary Size=256k2025.08 | 0.0294 | 0.12 | 48.3 | 3.696 | 0.46 | 0.056 | 0.529 | 0.598 | |
| PA-BPE (window+hybrid)Vocabulary Size=256k2025.08 | 0.0294 | 0.1217 | 49 | 3.709 | 0.46 | 0.03 | 0.524 | 0.59 | |
| PA-BPE (hybrid-127k)Vocabulary Size=127k2025.08 | 0.0294 | 0.1188 | 47.9 | 3.722 | 0.46 | 0.027 | 0.521 | 0.59 | |
| PA-BPE (window)Vocabulary Size=256k2025.08 | 0.0292 | 0.1203 | 48.9 | 3.725 | 0.47 | 0.032 | 0.514 | 0.58 | |
| PA-BPE (hybrid)Vocabulary Size=256k2025.08 | 0.0291 | 0.1157 | 47.1 | 3.739 | 0.47 | 0.026 | 0.513 | 0.582 | |
| PA-BPEVocabulary Size=256k2025.08 | 0.0288 | 0.1143 | 47 | 3.752 | 0.47 | 0.028 | 0.501 | 0.57 | |
| PA-BPE (window+hybrid)Vocabulary Size=128k2025.08 | 0.0266 | 0.0761 | 67.9 | 4.127 | 0.49 | 0.031 | 0.43 | 0.517 | |
| PA-BPE (hybrid)Vocabulary Size=128k2025.08 | 0.0264 | 0.0737 | 66.3 | 4.116 | 0.5 | 0.027 | 0.426 | 0.515 | |
| BPEVocabulary Size=128k2025.08 | 0.0262 | 0.0718 | 65.1 | 4.178 | 0.5 | 0.067 | 0.434 | 0.527 | |
| PA-BPE (window)Vocabulary Size=128k2025.08 | 0.0262 | 0.0749 | 67.9 | 4.152 | 0.5 | 0.029 | 0.417 | 0.507 | |
| PA-BPEVocabulary Size=128k2025.08 | 0.026 | 0.0727 | 66.4 | 4.138 | 0.5 | 0.027 | 0.418 | 0.508 | |
| UnigramLMVocabulary Size=128k2025.08 | 0.0193 | 0.0457 | 56.3 | 4.536 | 0.29 | 0.095 | 0.147 | 0.262 |