List your company

BPE Tokenization on Chinese

1.59Speedup FactorIncremental BPE
0.90361.08181.261.4382May 29, 2026

Evaluation results

MethodLinks
Incremental BPETokenizer Model=CL100K, Library=tiktoken2026.051.59
Incremental BPETokenizer Model=O200K, Library=tiktoken2026.051.46
Incremental BPETokenizer Model=P50K, Library=tiktoken2026.051.35
Incremental BPETokenizer Model=R50K, Library=tiktoken2026.051.35
Incremental BPETokenizer Model=CodeLlama, Library=tokenizers2026.051.1
Incremental BPETokenizer Model=GPT-OSS, Library=tokenizers2026.051.08
Incremental BPETokenizer Model=Qwen-3, Library=tokenizers2026.051.04
Incremental BPETokenizer Model=Mistral-3, Library=tokenizers2026.051.04
Incremental BPETokenizer Model=Llama-3.1*, Library=tokenizers2026.051.03
Incremental BPETokenizer Model=Ouro, Library=tokenizers2026.051.02
Incremental BPETokenizer Model=Llama-4, Library=tokenizers2026.051.01
Incremental BPETokenizer Model=DeepSeek-3.2, Library=tokenizers2026.050.93