List your company
ResearchFormerly WizWand

Incremental BPE Tokenization on Code

1.369End-to-end CPU Time (s)Ours (inc)
1.338921.541961.7451.94804May 29, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Ours (inc)Model=R50K2026.051.369
Ours (inc)Model=CL100K2026.051.422
via backtrackingModel=R50K2026.051.431
via backtrackingModel=CL100K2026.051.441
BaselineModel=R50K2026.051.501
Ours (eager)Model=R50K2026.051.51
Ours (eager)Model=CL100K2026.051.515
BaselineModel=CL100K2026.051.535
via tableModel=R50K2026.051.566
via tableModel=CL100K2026.051.579
Ours (inc)Model=O200K2026.052.032
Ours (eager)Model=O200K2026.052.081
BaselineModel=O200K2026.052.082
via backtrackingModel=O200K2026.052.087
via tableModel=O200K2026.052.121