Intrinsic Tokenization Evaluation on ClimbMix (held-out)
99Vocabulary UtilizationDet
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DetVocabulary Size=8k2026.05 | 99 | 0.0056 | 10.61 | 7.02 | 1,141.3 | 4.09 | 62.3 | 0.0014 | |
| BiasVocabulary Size=8k2026.05 | 99 | 0.0054 | 10.69 | 7.1 | 1,171 | 3.93 | 64.6 | 0.0013 | |
| DetVocabulary Size=16k2026.05 | 98.8 | 0.0123 | 10.9 | 6.83 | 1,833.8 | 4.46 | 57 | 0.0015 | |
| BiasVocabulary Size=16k2026.05 | 98.8 | 0.012 | 10.95 | 6.88 | 1,860.9 | 4.36 | 58.3 | 0.0015 | |
| IntVocabulary Size=16k2026.05 | 98.8 | 0.0082 | 10.18 | 6.99 | 1,033.6 | 4.03 | 63.6 | 0.0014 | |
| BPEVocabulary Size=8k2026.05 | 98.6 | 0.0055 | 10.61 | 7.09 | 1,121.5 | 3.99 | 63.8 | 0.0014 | |
| IntVocabulary Size=8k2026.05 | 98.6 | 0.0031 | 9.31 | 6.9 | 492.3 | 3.39 | 75.6 | 0.0011 | |
| BPEVocabulary Size=16k2026.05 | 98.1 | 0.012 | 10.9 | 6.88 | 1,780.4 | 4.38 | 58.1 | 0.0015 | |
| IntVocabulary Size=32k2026.05 | 93.4 | 0.0195 | 10.82 | 6.79 | 1,975.1 | 4.55 | 56 | 0.0015 | |
| BiasVocabulary Size=32k2026.05 | 92.6 | 0.0242 | 11.06 | 6.72 | 2,609.1 | 4.68 | 54.4 | 0.0016 | |
| DetVocabulary Size=32k2026.05 | 92.4 | 0.0244 | 11.03 | 6.7 | 2,586.1 | 4.72 | 53.9 | 0.0016 | |
| BPEVocabulary Size=32k2026.05 | 92.1 | 0.024 | 11.04 | 6.73 | 2,516.8 | 4.67 | 54.5 | 0.0016 | |
| IntVocabulary Size=64k2026.05 | 73.8 | 0.0336 | 10.98 | 6.66 | 2,760.2 | 4.8 | 53 | 0.0016 | |
| BiasVocabulary Size=64k2026.05 | 71.5 | 0.0388 | 11.07 | 6.63 | 3,226.2 | 4.85 | 52.4 | 0.0017 | |
| DetVocabulary Size=64k2026.05 | 71.2 | 0.0389 | 11.05 | 6.62 | 3,210.4 | 4.88 | 52.1 | 0.0017 | |
| BPEVocabulary Size=64k2026.05 | 70.7 | 0.0384 | 11.07 | 6.64 | 3,138 | 4.85 | 52.4 | 0.0017 | |
| IntVocabulary Size=128k2026.05 | 45.8 | 0.0463 | 11.03 | 6.58 | 3,410.6 | 4.96 | 51.3 | 0.0017 | |
| BiasVocabulary Size=128k2026.05 | 44 | 0.049 | 11.04 | 6.58 | 3,566.7 | 4.97 | 51.2 | 0.0017 | |
| DetVocabulary Size=128k2026.05 | 43.8 | 0.0489 | 11.03 | 6.58 | 3,550.8 | 4.98 | 51.1 | 0.0017 | |
| BPEVocabulary Size=128k2026.05 | 43.3 | 0.0481 | 11.05 | 6.59 | 3,504.5 | 4.96 | 51.3 | 0.0017 | |
| IntVocabulary Size=256k2026.05 | 25.2 | 0.0514 | 11 | 6.56 | 3,563 | 5.01 | 50.7 | 0.0017 | |
| BiasVocabulary Size=256k2026.05 | 23.7 | 0.0534 | 11.01 | 6.55 | 3,671.1 | 5.02 | 50.6 | 0.0017 | |
| DetVocabulary Size=256k2026.05 | 23.6 | 0.0532 | 11.01 | 6.55 | 3,660.9 | 5.03 | 50.6 | 0.0017 | |
| BPEVocabulary Size=256k2026.05 | 23.3 | 0.0525 | 11.02 | 6.56 | 3,635.4 | 5.02 | 50.7 | 0.0017 |