Tokenizer compression on C4 ko
1.72C/T RatioGPT-OSS
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-OSSVocabulary Size=200k2026.02 | 1.72 | |
| Llama 3Vocabulary Size=128k2026.02 | 1.68 | |
| DeepSeek R1Vocabulary Size=128k2026.02 | 1.47 | |
| Qwen 3Vocabulary Size=152k2026.02 | 1.47 | |
| TrinityVocabulary Size=200k2026.02 | 1.26 |