List your company

Incremental BPE Tokenization on English

0.925Median End-to-end CPU Time (s)Ours (inc)
0.888441.135221.3821.62878May 29, 2026

Evaluation results

MethodLinks
Ours (inc)Model=R50K2026.050.925
BaselineModel=R50K2026.050.932
via backtrackingModel=R50K2026.050.948
Ours (eager)Model=R50K2026.050.989
via tableModel=R50K2026.051.004
Ours (inc)Model=CL100K2026.051.064
BaselineModel=CL100K2026.051.068
via backtrackingModel=CL100K2026.051.09
Ours (eager)Model=CL100K2026.051.135
via tableModel=CL100K2026.051.14
Ours (inc)Model=O200K2026.051.739
BaselineModel=O200K2026.051.768
via backtrackingModel=O200K2026.051.782
Ours (eager)Model=O200K2026.051.818
via tableModel=O200K2026.051.839