Tokenization on WikiText-103
1.92Latency (ms)tiktoken
Evaluation results
| Method | Links | |
|---|---|---|
| tiktokenSequence Length=2562026.03 | 1.92 | |
| HF GPT2Sequence Length=2562026.03 | 4.21 | |
| GPU-OptSequence Length=2562026.03 | 4.98 | |
| GPU-BaseSequence Length=2562026.03 | 5.77 | |
| tiktokenSequence Length=1K2026.03 | 6.32 | |
| GPU-OptSequence Length=1K2026.03 | 9.42 | |
| GPU-BaseSequence Length=1K2026.03 | 10.9 | |
| HF GPT2Sequence Length=1K2026.03 | 12.7 | |
| CPU-BPESequence Length=2562026.03 | 12.84 | |
| GPU-OptSequence Length=4K2026.03 | 14.5 | |
| GPU-BaseSequence Length=4K2026.03 | 18.3 | |
| tiktokenSequence Length=4K2026.03 | 22.1 | |
| GPU-OptSequence Length=16K2026.03 | 24.1 | |
| GPU-BaseSequence Length=16K2026.03 | 29.5 | |
| CPU-BPESequence Length=1K2026.03 | 48.5 | |
| HF GPT2Sequence Length=4K2026.03 | 49.4 | |
| GPU-OptSequence Length=131K2026.03 | 53.4 | |
| GPU-BaseSequence Length=131K2026.03 | 89.3 | |
| tiktokenSequence Length=16K2026.03 | 91.3 | |
| CPU-BPESequence Length=4K2026.03 | 189.2 | |
| HF GPT2Sequence Length=16K2026.03 | 201 | |
| tiktokenSequence Length=131K2026.03 | 598.1 | |
| CPU-BPESequence Length=16K2026.03 | 749.9 | |
| HF GPT2Sequence Length=131K2026.03 | 1,673 | |
| CPU-BPESequence Length=131K2026.03 | 6,143 |