Tokenization on Multilingual code evaluation set
3.64Compression Ratio (CR)SA-BPE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SA-BPETokenizer=SA-BPE, Priority criterion=F · log R · L2026.04 | 3.64 | 31,567 | 5.86 | 0 | |
| BPETokenizer=BPE2026.04 | 3.62 | 29,240 | 6.23 | 0 | |
| PickyBPETokenizer=PickyBPE2026.04 | 3.61 | 28,673 | 6.22 | 1,620 | |
| BoundlessBPETokenizer=BoundlessBPE2026.04 | 3.56 | 29,375 | 6.7 | 1,680 |