List your company

Incremental BPE Tokenization on Chinese

0.815End-to-end CPU Time (s)Ours (inc)
0.778321.025911.27351.52109May 29, 2026

Evaluation results

MethodLinks
Ours (inc)Model=CL100K2026.050.815
Ours (inc)Model=R50K2026.050.897
Ours (inc)Model=O200K2026.051.047
via backtrackingModel=CL100K2026.051.048
Ours (eager)Model=CL100K2026.051.05
Ours (eager)Model=R50K2026.051.117
via backtrackingModel=R50K2026.051.18
via tableModel=R50K2026.051.243
BaselineModel=R50K2026.051.244
via tableModel=CL100K2026.051.254
via backtrackingModel=O200K2026.051.275
Ours (eager)Model=O200K2026.051.308
BaselineModel=CL100K2026.051.374
BaselineModel=O200K2026.051.729
via tableModel=O200K2026.051.732