BPE Tokenization on Chinese
1.59Speedup FactorIncremental BPE
Evaluation Results
| Method | Links | |
|---|---|---|
| Incremental BPETokenizer Model=CL100K, Library=tiktoken2026.05 | 1.59 | |
| Incremental BPETokenizer Model=O200K, Library=tiktoken2026.05 | 1.46 | |
| Incremental BPETokenizer Model=P50K, Library=tiktoken2026.05 | 1.35 | |
| Incremental BPETokenizer Model=R50K, Library=tiktoken2026.05 | 1.35 | |
| Incremental BPETokenizer Model=CodeLlama, Library=tokenizers2026.05 | 1.1 | |
| Incremental BPETokenizer Model=GPT-OSS, Library=tokenizers2026.05 | 1.08 | |
| Incremental BPETokenizer Model=Qwen-3, Library=tokenizers2026.05 | 1.04 | |
| Incremental BPETokenizer Model=Mistral-3, Library=tokenizers2026.05 | 1.04 | |
| Incremental BPETokenizer Model=Llama-3.1*, Library=tokenizers2026.05 | 1.03 | |
| Incremental BPETokenizer Model=Ouro, Library=tokenizers2026.05 | 1.02 | |
| Incremental BPETokenizer Model=Llama-4, Library=tokenizers2026.05 | 1.01 | |
| Incremental BPETokenizer Model=DeepSeek-3.2, Library=tokenizers2026.05 | 0.93 |