Incremental BPE Tokenization on Code
1.369End-to-end CPU Time (s)Ours (inc)
Evaluation Results
| Method | Links | |
|---|---|---|
| Ours (inc)Model=R50K2026.05 | 1.369 | |
| Ours (inc)Model=CL100K2026.05 | 1.422 | |
| via backtrackingModel=R50K2026.05 | 1.431 | |
| via backtrackingModel=CL100K2026.05 | 1.441 | |
| BaselineModel=R50K2026.05 | 1.501 | |
| Ours (eager)Model=R50K2026.05 | 1.51 | |
| Ours (eager)Model=CL100K2026.05 | 1.515 | |
| BaselineModel=CL100K2026.05 | 1.535 | |
| via tableModel=R50K2026.05 | 1.566 | |
| via tableModel=CL100K2026.05 | 1.579 | |
| Ours (inc)Model=O200K2026.05 | 2.032 | |
| Ours (eager)Model=O200K2026.05 | 2.081 | |
| BaselineModel=O200K2026.05 | 2.082 | |
| via backtrackingModel=O200K2026.05 | 2.087 | |
| via tableModel=O200K2026.05 | 2.121 |