BPE Tokenization on Code
2.88Speedup FactorIncremental BPE
Evaluation results
| Method | Links | |
|---|---|---|
| Incremental BPETokenizer Model=CodeLlama, Library=tokenizers2026.05 | 2.88 | |
| Incremental BPETokenizer Model=Qwen-3, Library=tokenizers2026.05 | 1.08 | |
| Incremental BPETokenizer Model=P50K, Library=tiktoken2026.05 | 1.07 | |
| Incremental BPETokenizer Model=R50K, Library=tiktoken2026.05 | 1.05 | |
| Incremental BPETokenizer Model=CL100K, Library=tiktoken2026.05 | 1.04 | |
| Incremental BPETokenizer Model=DeepSeek-3.2, Library=tokenizers2026.05 | 1.03 | |
| Incremental BPETokenizer Model=Ouro, Library=tokenizers2026.05 | 1.02 | |
| Incremental BPETokenizer Model=Llama-3.1*, Library=tokenizers2026.05 | 1.02 | |
| Incremental BPETokenizer Model=GPT-OSS, Library=tokenizers2026.05 | 1.01 | |
| Incremental BPETokenizer Model=Llama-4, Library=tokenizers2026.05 | 1 | |
| Incremental BPETokenizer Model=O200K, Library=tiktoken2026.05 | 1 | |
| Incremental BPETokenizer Model=Mistral-3, Library=tokenizers2026.05 | 0.99 |