Tokenizer Compression on C4 fr
3.34Bits per Token (B/T)Qwen 3
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen 3Vocabulary Size=152k2026.02 | 3.34 | |
| Llama 3Vocabulary Size=128k2026.02 | 3.49 | |
| DeepSeek R1Vocabulary Size=128k2026.02 | 3.53 | |
| GPT-OSSVocabulary Size=200k2026.02 | 3.97 | |
| TrinityVocabulary Size=200k2026.02 | 3.98 |