Tokenization Intrinsic Evaluation on FLORES+ 30-lang
0.027Compression RatePA-BPE (ratios)
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| PA-BPE (ratios)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=ratios2025.08 | 0.027 | 0.0737 | 64.9 | 4.398 | 0.49 | 0.04 | 0.532 | 0.669 | |
| BPEVocabulary Size=128k, Training Data=unbalanced 30-lang dataset2025.08 | 0.0275 | 0.0777 | 67 | 3.99 | 0.49 | 0.064 | 0.537 | 0.659 | |
| PA-BPEVocabulary Size=128k, Training Data=unbalanced 30-lang dataset2025.08 | 0.0276 | 0.0819 | 70.4 | 4.08 | 0.49 | 0.007 | 0.539 | 0.671 | |
| PA-BPE (window)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=window2025.08 | 0.0278 | 0.0838 | 71.6 | 4.042 | 0.48 | 0.009 | 0.546 | 0.677 | |
| PA-BPE (hybrid)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=hybrid2025.08 | 0.0278 | 0.0817 | 69.8 | 4.056 | 0.49 | 0.015 | 0.541 | 0.667 | |
| PA-BPE (hybrid+window)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=hybrid+window2025.08 | 0.0279 | 0.0832 | 70.8 | 4.029 | 0.48 | 0.019 | 0.546 | 0.67 |