Pretokenization Offsets Calculation on LLaMA-3 Tokenizer Evaluation Data
46.037Throughput (Mbps)Peek2
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Peek2Implementation=peek2, Task=llama3-batch2026.01 | 46.037 | 140.9445 | 0.5818 | |
| Original (tiktoken)Implementation=original, Task=llama3-batch2026.01 | 41.4111 | 156.6892 | 0.7073 | |
| Peek2Implementation=peek2, Task=llama3-offsets2026.01 | 39.6799 | 163.5253 | 1.2281 | |
| Original (tiktoken)Implementation=original, Task=llama3-offsets2026.01 | 37.1597 | 174.6158 | 0.6362 |