Incremental BPE Tokenization on English
0.925Median End-to-end CPU Time (s)Ours (inc)
Evaluation Results
| Method | Links | |
|---|---|---|
| Ours (inc)Model=R50K2026.05 | 0.925 | |
| BaselineModel=R50K2026.05 | 0.932 | |
| via backtrackingModel=R50K2026.05 | 0.948 | |
| Ours (eager)Model=R50K2026.05 | 0.989 | |
| via tableModel=R50K2026.05 | 1.004 | |
| Ours (inc)Model=CL100K2026.05 | 1.064 | |
| BaselineModel=CL100K2026.05 | 1.068 | |
| via backtrackingModel=CL100K2026.05 | 1.09 | |
| Ours (eager)Model=CL100K2026.05 | 1.135 | |
| via tableModel=CL100K2026.05 | 1.14 | |
| Ours (inc)Model=O200K2026.05 | 1.739 | |
| BaselineModel=O200K2026.05 | 1.768 | |
| via backtrackingModel=O200K2026.05 | 1.782 | |
| Ours (eager)Model=O200K2026.05 | 1.818 | |
| via tableModel=O200K2026.05 | 1.839 |