Tokenizer Compression on C4 ja
1.6C/TDeepSeek R1
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepSeek R1Vocabulary Size=128k2026.02 | 1.6 | |
| Llama 3Vocabulary Size=128k2026.02 | 1.6 | |
| Qwen 3Vocabulary Size=152k2026.02 | 1.59 | |
| GPT-OSSVocabulary Size=200k2026.02 | 1.54 | |
| TrinityVocabulary Size=200k2026.02 | 1.39 |