BPE Tokenization on English
3.13Speedup FactorIncremental BPE
Evaluation Results
| Method | Links | |
|---|---|---|
| Incremental BPETokenizer Model=CodeLlama, Library=tokenizers2026.05 | 3.13 | |
| Incremental BPETokenizer Model=Qwen-3, Library=tokenizers2026.05 | 1.05 | |
| Incremental BPETokenizer Model=DeepSeek-3.2, Library=tokenizers2026.05 | 1.01 | |
| Incremental BPETokenizer Model=Llama-4, Library=tokenizers2026.05 | 1.01 | |
| Incremental BPETokenizer Model=Ouro, Library=tokenizers2026.05 | 1 | |
| Incremental BPETokenizer Model=GPT-OSS, Library=tokenizers2026.05 | 1 | |
| Incremental BPETokenizer Model=Mistral-3, Library=tokenizers2026.05 | 1 | |
| Incremental BPETokenizer Model=Llama-3.1*, Library=tokenizers2026.05 | 0.99 | |
| Incremental BPETokenizer Model=O200K, Library=tiktoken2026.05 | 0.99 | |
| Incremental BPETokenizer Model=P50K, Library=tiktoken2026.05 | 0.97 | |
| Incremental BPETokenizer Model=R50K, Library=tiktoken2026.05 | 0.96 | |
| Incremental BPETokenizer Model=CL100K, Library=tiktoken2026.05 | 0.96 |