Tokenizer Compression on C4 en
4.64Bits per TokenQwen 3
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen 3Vocabulary Size=152k2026.02 | 4.64 | |
| DeepSeek R1Vocabulary Size=128k2026.02 | 4.7 | |
| Llama 3Vocabulary Size=128k2026.02 | 4.73 | |
| GPT-OSSVocabulary Size=200k2026.02 | 4.79 | |
| TrinityVocabulary Size=200k2026.02 | 4.84 |