List your company

BPE Tokenization on English

3.13Speedup FactorIncremental BPE
0.87321.45912.0452.6309May 29, 2026

Evaluation results

MethodLinks
Incremental BPETokenizer Model=CodeLlama, Library=tokenizers2026.053.13
Incremental BPETokenizer Model=Qwen-3, Library=tokenizers2026.051.05
Incremental BPETokenizer Model=DeepSeek-3.2, Library=tokenizers2026.051.01
Incremental BPETokenizer Model=Llama-4, Library=tokenizers2026.051.01
Incremental BPETokenizer Model=Ouro, Library=tokenizers2026.051
Incremental BPETokenizer Model=GPT-OSS, Library=tokenizers2026.051
Incremental BPETokenizer Model=Mistral-3, Library=tokenizers2026.051
Incremental BPETokenizer Model=Llama-3.1*, Library=tokenizers2026.050.99
Incremental BPETokenizer Model=O200K, Library=tiktoken2026.050.99
Incremental BPETokenizer Model=P50K, Library=tiktoken2026.050.97
Incremental BPETokenizer Model=R50K, Library=tiktoken2026.050.96
Incremental BPETokenizer Model=CL100K, Library=tiktoken2026.050.96