Tokenization Fidelity and Throughput Analysis on Penn Treebank (PTB)
30.44Throughput (Bytes/sec)pllama1B ◦ fα ◦ fptb
Evaluation results
| Method | Links | ||
|---|---|---|---|
| pllama1B ◦ fα ◦ fptbthreshold (tau)=1e-12026.03 | 30.44 | 0.003 | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=1e-12026.03 | 26.65 | 0.0039 | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=1e-12026.03 | 19.25 | 0.0025 | |
| pllama1B ◦ fα ◦ fptbthreshold (tau)=3e-22026.03 | 17.45 | 0.0011 | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=3e-22026.03 | 17.03 | 0.0014 | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=3e-22026.03 | 13.38 | 0.001 | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=1e-22026.03 | 10.79 | 0.0006 | |
| pllama1B ◦ fα ◦ fptbthreshold (tau)=1e-22026.03 | 10.54 | 0.0004 | |
| pphi4 ◦ fα ◦ fptbthreshold (tau)=1e-12026.03 | 10.07 | 0.0046 | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=1e-22026.03 | 9.49 | 0.0006 | |
| pllama1B ◦ fα ◦ fptbthreshold (tau)=3e-32026.03 | 8.3 | 0.0001 | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=3e-32026.03 | 5.9 | 0.0001 | |
| pphi4 ◦ fα ◦ fptbthreshold (tau)=3e-22026.03 | 5.87 | 0.0038 | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=3e-32026.03 | 4.52 | 0.0002 | |
| pllama1B ◦ fα ◦ fptbthreshold (tau)=1e-32026.03 | 4.13 | 0.0001 | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=1e-32026.03 | 3.29 | 0.0001 | |
| pphi4 ◦ fα ◦ fptbthreshold (tau)=3e-32026.03 | 2.95 | 0.0004 | |
| pphi4 ◦ fα ◦ fptbthreshold (tau)=1e-22026.03 | 2.94 | 0.0032 | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=1e-32026.03 | 2.01 | 0.0001 | |
| pllama1B ◦ fα ◦ fptbthreshold (tau)=3e-42026.03 | 1.67 | 0 | |
| pphi4 ◦ fα ◦ fptbthreshold (tau)=1e-32026.03 | 1.54 | 0.0004 | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=3e-42026.03 | 1.42 | 0 | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=3e-42026.03 | 0.81 | 0 | |
| pllama1B ◦ fα ◦ fptbthreshold (tau)=1e-42026.03 | 0.7 | — | |
| pllama8B ◦ fα ◦ fptbthreshold (tau)=1e-42026.03 | 0.56 | — | |
| pphi4 ◦ fα ◦ fptbthreshold (tau)=3e-42026.03 | 0.48 | — | |
| pgpt2 ◦ fα ◦ fptbthreshold (tau)=1e-42026.03 | 0.2 | — |