Pretokenization Encoding on LLaMA-3 Tokenizer Evaluation Data
7.0056Throughput (Mbps)Peek2
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Peek2Implementation=peek2, Task=llama3-encode2026.01 | 7.0056 | 926.2106 | 1.2306 | |
| Original (tiktoken)Implementation=original, Task=llama3-encode2026.01 | 6.2071 | 1,045.3602 | 0.8256 |