Subword-to-byte distribution approximation on reference distribution token-to-byte 2025a (test)
0Average JSD / bytepgpt2 ◦ fα
Evaluation results
| Method | Links | ||
|---|---|---|---|
| pgpt2 ◦ fαthreshold (τ)=3e-5, beam size (K)=322026.03 | 0 | 0.21 | |
| pgpt2 ◦ fαthreshold (τ)=1e-5, beam size (K)=322026.03 | 0 | 0.19 | |
| pgpt2 ◦ fαthreshold (τ)=1e-4, beam size (K)=322026.03 | 0 | 0.65 | |
| pllama8B ◦ fαthreshold (τ)=1e-5, beam size (K)=322026.03 | 0.0001 | 1.4 | |
| pllama1B ◦ fαthreshold (τ)=1e-5, beam size (K)=322026.03 | 0.0001 | 0.62 | |
| pllama8B ◦ fαthreshold (τ)=3e-5, beam size (K)=322026.03 | 0.0001 | 2.38 | |
| pgpt2 ◦ fαthreshold (τ)=3e-4, beam size (K)=322026.03 | 0.0001 | 1.97 | |
| pphi4 ◦ fαthreshold (τ)=3e-5, beam size (K)=322026.03 | 0.0001 | 1.3 | |
| pllama1B ◦ fαthreshold (τ)=3e-5, beam size (K)=322026.03 | 0.0001 | 1.2 | |
| pphi4 ◦ fαthreshold (τ)=1e-5, beam size (K)=322026.03 | 0.0001 | 0.74 | |
| pllama1B ◦ fαthreshold (τ)=1e-4, beam size (K)=322026.03 | 0.0001 | 2.15 | |
| pllama8B ◦ fαthreshold (τ)=1e-4, beam size (K)=322026.03 | 0.0001 | 4.03 | |
| pphi4 ◦ fαthreshold (τ)=1e-4, beam size (K)=322026.03 | 0.0001 | 2.19 | |
| pllama1B ◦ fαthreshold (τ)=3e-4, beam size (K)=322026.03 | 0.0001 | 5.9 | |
| pllama8B ◦ fαthreshold (τ)=3e-4, beam size (K)=322026.03 | 0.0001 | 9.08 | |
| pphi4 ◦ fαthreshold (τ)=3e-4, beam size (K)=322026.03 | 0.0002 | 4.87 | |
| pllama1B ◦ fαthreshold (τ)=1e-3, beam size (K)=322026.03 | 0.0004 | 11.86 | |
| pllama8B ◦ fαthreshold (τ)=1e-3, beam size (K)=322026.03 | 0.0005 | 15.73 | |
| pphi4 ◦ fαthreshold (τ)=1e-3, beam size (K)=322026.03 | 0.0009 | 8.75 | |
| pllama1B ◦ fαthreshold (τ)=3e-3, beam size (K)=322026.03 | 0.0011 | 22.09 | |
| pllama8B ◦ fαthreshold (τ)=3e-3, beam size (K)=322026.03 | 0.0013 | 26.71 | |
| pgpt2 ◦ fαthreshold (τ)=1e-3, beam size (K)=322026.03 | 0.0014 | 6.21 | |
| pphi4 ◦ fαthreshold (τ)=3e-3, beam size (K)=322026.03 | 0.0018 | 14.8 | |
| pgpt2 ◦ fαthreshold (τ)=3e-3, beam size (K)=322026.03 | 0.0025 | 10.48 | |
| pllama8B ◦ fαthreshold (τ)=1e-2, beam size (K)=322026.03 | 0.0035 | 45.47 | |
| pllama1B ◦ fαthreshold (τ)=1e-2, beam size (K)=322026.03 | 0.0042 | 40.45 | |
| pphi4 ◦ fαthreshold (τ)=1e-2, beam size (K)=322026.03 | 0.0046 | 22.85 | |
| pgpt2 ◦ fαthreshold (τ)=1e-2, beam size (K)=322026.03 | 0.0067 | 25.13 | |
| pllama8B ◦ fαthreshold (τ)=3e-2, beam size (K)=322026.03 | 0.0088 | 66.41 | |
| pphi4 ◦ fαthreshold (τ)=3e-2, beam size (K)=322026.03 | 0.0097 | 36.2 | |
| pllama1B ◦ fαthreshold (τ)=3e-2, beam size (K)=322026.03 | 0.012 | 72.23 | |
| pgpt2 ◦ fαthreshold (τ)=3e-2, beam size (K)=322026.03 | 0.019 | 44.68 | |
| pllama8B ◦ fαthreshold (τ)=1e-1, beam size (K)=322026.03 | 0.022 | 79.03 | |
| pphi4 ◦ fαthreshold (τ)=1e-1, beam size (K)=322026.03 | 0.025 | 41.04 | |
| pllama1B ◦ fαthreshold (τ)=1e-1, beam size (K)=322026.03 | 0.028 | 84.21 | |
| pgpt2 ◦ fαthreshold (τ)=1e-1, beam size (K)=322026.03 | 0.04 | 70.95 |