Tokenizer Compression on C4 (zh subset)
1.54C/T RatioDeepSeek R1
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeek R1Vocabulary Size=128k2026.02 | 1.54 | |
| GPT-OSSVocabulary Size=200k2026.02 | 1.47 | |
| Qwen 3Vocabulary Size=152k2026.02 | 1.45 | |
| TrinityVocabulary Size=200k2026.02 | 1.38 | |
| Llama 3Vocabulary Size=128k2026.02 | 1.29 |