Incremental BPE Tokenization on Chinese
0.815End-to-end CPU Time (s)Ours (inc)
Evaluation Results
| Method | Links | |
|---|---|---|
| Ours (inc)Model=CL100K2026.05 | 0.815 | |
| Ours (inc)Model=R50K2026.05 | 0.897 | |
| Ours (inc)Model=O200K2026.05 | 1.047 | |
| via backtrackingModel=CL100K2026.05 | 1.048 | |
| Ours (eager)Model=CL100K2026.05 | 1.05 | |
| Ours (eager)Model=R50K2026.05 | 1.117 | |
| via backtrackingModel=R50K2026.05 | 1.18 | |
| via tableModel=R50K2026.05 | 1.243 | |
| BaselineModel=R50K2026.05 | 1.244 | |
| via tableModel=CL100K2026.05 | 1.254 | |
| via backtrackingModel=O200K2026.05 | 1.275 | |
| Ours (eager)Model=O200K2026.05 | 1.308 | |
| BaselineModel=CL100K2026.05 | 1.374 | |
| BaselineModel=O200K2026.05 | 1.729 | |
| via tableModel=O200K2026.05 | 1.732 |