Intrinsic Tokenization on FLORES+ (test)
69.2Vocabulary UtilisationBias
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| BiasVocabulary Size=8k2026.05 | 69.2 | 0.0093 | 6.79 | 4.29 | 156.7 | 1.96 | 121.9 | 0.0082 | |
| BPEVocabulary Size=8k2026.05 | 67.7 | 0.009 | 6.75 | 4.34 | 131.1 | 1.91 | 123.5 | 0.0081 | |
| DetVocabulary Size=8k2026.05 | 63.8 | 0.0085 | 6.73 | 4.31 | 127.8 | 1.93 | 123.4 | 0.0081 | |
| IntVocabulary Size=8k2026.05 | 62 | 0.0051 | 6.41 | 4.54 | 57.6 | 1.52 | 136.7 | 0.0073 | |
| BiasVocabulary Size=16k2026.05 | 48.1 | 0.0154 | 7.37 | 4.84 | 233.9 | 2.19 | 102.9 | 0.0097 | |
| BPEVocabulary Size=16k2026.05 | 46.5 | 0.0142 | 7.25 | 4.9 | 192.2 | 2.11 | 107.9 | 0.0093 | |
| DetVocabulary Size=16k2026.05 | 44.3 | 0.0138 | 7.29 | 4.86 | 196.3 | 2.16 | 105.8 | 0.0095 | |
| IntVocabulary Size=16k2026.05 | 42.7 | 0.0084 | 6.76 | 4.8 | 89.1 | 1.77 | 124.4 | 0.008 | |
| BiasVocabulary Size=32k2026.05 | 29.4 | 0.0216 | 7.85 | 5.27 | 322.5 | 2.39 | 89.4 | 0.0112 | |
| BPEVocabulary Size=32k2026.05 | 28.7 | 0.0201 | 7.71 | 5.27 | 272.5 | 2.3 | 93.5 | 0.0107 | |
| DetVocabulary Size=32k2026.05 | 27.7 | 0.02 | 7.76 | 5.25 | 285.4 | 2.36 | 91 | 0.011 | |
| IntVocabulary Size=32k2026.05 | 26.6 | 0.0132 | 7.22 | 5.13 | 153.1 | 2.06 | 108.8 | 0.0092 | |
| BiasVocabulary Size=64k2026.05 | 17.5 | 0.0334 | 8.91 | 6.37 | 519.6 | 2.65 | 69.1 | 0.0145 | |
| BPEVocabulary Size=64k2026.05 | 16.7 | 0.0301 | 8.79 | 6.68 | 432.4 | 2.57 | 72.7 | 0.0138 | |
| DetVocabulary Size=64k2026.05 | 16.6 | 0.0313 | 8.84 | 6.33 | 475.6 | 2.63 | 69.9 | 0.0143 | |
| IntVocabulary Size=64k2026.05 | 15.8 | 0.0195 | 7.82 | 5.55 | 245.6 | 2.3 | 90.1 | 0.0111 | |
| BiasVocabulary Size=128k2026.05 | 10 | 0.0459 | 9.74 | 6.85 | 779.8 | 2.92 | 57.6 | 0.0174 | |
| DetVocabulary Size=128k2026.05 | 9.7 | 0.0442 | 9.7 | 6.83 | 742.1 | 2.91 | 57.9 | 0.0173 | |
| BPEVocabulary Size=128k2026.05 | 9.4 | 0.0405 | 9.52 | 7.25 | 632.3 | 2.83 | 61 | 0.0164 | |
| IntVocabulary Size=128k2026.05 | 9.3 | 0.0321 | 8.72 | 6.37 | 469.7 | 2.66 | 69.6 | 0.0144 | |
| BiasVocabulary Size=256k2026.05 | 5.9 | 0.0624 | 10.46 | 7.6 | 1,140.2 | 3.19 | 49.9 | 0.02 | |
| DetVocabulary Size=256k2026.05 | 5.8 | 0.0608 | 10.41 | 7.56 | 1,107 | 3.19 | 49.8 | 0.0201 | |
| BPEVocabulary Size=256k2026.05 | 5.4 | 0.0548 | 10.28 | 7.71 | 951 | 3.1 | 51.9 | 0.0193 | |
| IntVocabulary Size=256k2026.05 | 5.2 | 0.0391 | 9.09 | 6.74 | 590.3 | 2.8 | 63.9 | 0.0157 |