Language Modeling on The Pile (test)
9.213PPL (The Pile Test)GPT-Neo 1.3B
Evaluation results
| Method | Links | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-Neo 1.3Btheta=0, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 9.213 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBudget=24x, Configuration=(24 ⊗ 1), Params=24x, FLOPs=24x2026.02 | 9.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 1.3Btheta=0.5, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 9.758 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBudget=12x, Configuration=(12 ⊗ 1), Params=12x, FLOPs=12x2026.02 | 9.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 1.3Btheta=0.75, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 10.267 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DAG-MoE-lNumber of Parameters=699M2026.05 | 10.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoopFormerBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 10.28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TMLTBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 10.38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MoE-lNumber of Parameters=699M2026.05 | 10.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TMLT-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 10.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 1.3Btheta=1, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 10.775 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base-LoopBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 10.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoopFormerBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 11.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBudget=6x, Configuration=(6 ⊗ 1), Params=6x, FLOPs=6x2026.02 | 11.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base-Loop-EE-ConsBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 11.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Naive-Loop-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 11.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Naive-Loop-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 11.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base-Loop-EE-ConsBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TMLT-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 12.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Naive-Loop-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 12.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoopFormerBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 14.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base-Loop-EE-ConsBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 15.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 125Mtheta=0, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 15.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TMLT-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 15.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 125Mtheta=1.25, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 16.601 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT2 1.5BPrefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 17.155 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 125Mtheta=1.5, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 17.499 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 125Mtheta=1.75, Prefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 19.691 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SAGE(Hybrid)Model Scale=1.3B2026.04 | 24.33 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.9 | — | — | |
| LionModel Scale=0.6B2026.04 | 26.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.2 | — | — | |
| SAGE(Hybrid)Model Scale=0.6B2026.04 | 26.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7 | — | — | |
| AdamWModel Scale=1.3B2026.04 | 27.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 9.8 | — | — | |
| LionModel Scale=1.3B2026.04 | 28.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.9 | — | — | |
| Lion-HybridModel Scale=1.3B2026.04 | 28.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.9 | — | — | |
| SinkGD(Hybrid)Model Scale=1.3B2026.04 | 28.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.9 | — | — | |
| Lion-HybridModel Scale=0.6B2026.04 | 28.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7 | — | — | |
| SAGE(Hybrid)Model Scale=270M2026.04 | 29.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.5 | — | — | |
| LionModel Scale=270M2026.04 | 30.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1 | — | — | |
| GPT2 124MPrefix Size=50, Suffix Size=50, Prompt Length=12023.05 | 30.323 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdamWModel Scale=0.6B2026.04 | 31.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.4 | — | — | |
| Lion-HybridModel Scale=270M2026.04 | 32.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.5 | — | — | |
| SinkGD(Hybrid)Model Scale=0.6B2026.04 | 32.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.5 | — | — | |
| SinkGD(Hybrid)Model Scale=270M2026.04 | 34.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.9 | — | — | |
| AdamWModel Scale=270M2026.04 | 37.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.1 | — | — | |
| APOLLOModel Scale=1.3B2026.04 | 39.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.6 | — | — | |
| APOLLOModel Scale=0.6B2026.04 | 44.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.1 | — | — | |
| APOLLOModel Scale=270M2026.04 | 52.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.2 | — | — | |
| SinkGD-PureModel Scale=1.3B2026.04 | 98.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | — | — | |
| SAGE-PureModel Scale=270M2026.04 | 116 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1 | — | — | |
| SinkGD-PureModel Scale=0.6B2026.04 | 117.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | — | — | |
| SAGE-PureModel Scale=0.6B2026.04 | 179 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.2 | — | — | |
| SinkGD-PureModel Scale=270M2026.04 | 192.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | — | — | |
| SAGE-PureModel Scale=1.3B2026.04 | 216 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.9 | — | — | |
| GLM-130B2022.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.634 | — | — | — | |
| Gopher2021.12 | — | 1.14 | 1.09 | 0.899 | 0.677 | 0.89 | 0.712 | 0.741 | 0.691 | 0.695 | 0.656 | 0.662 | 0.641 | 0.59 | 0.577 | 0.546 | 0.525 | 0.513 | 0.377 | — | — | — | — | |
| GPT-32022.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.742 | — | — | — | |
| GPT-32021.12 | — | 1.37 | 0.946 | 0.932 | 0.624 | 0.975 | 0.802 | 0.779 | 0.698 | 0.723 | 1.16 | 0.838 | 0.773 | 0.612 | 0.625 | 0.566 | 0.69 | 0.612 | 0.645 | — | — | — | — | |
| HullFTTime Budget (T)=1.75s, Average Selection Size (N*)=192026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7448 | 3.83 | |
| HullFTTime Budget (T)=2.0s, Average Selection Size (N*)=222026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7312 | 3.44 | |
| Jurassic2021.12 | — | 1.04 | 0.857 | 0.879 | — | 0.869 | 0.835 | — | 0.669 | 0.741 | 0.89 | 0.68 | 0.655 | 0.59 | 0.587 | 0.537 | 0.579 | 0.514 | 0.358 | — | — | — | — | |
| Jurassic-12022.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.65 | — | — | — | |
| kNNTime Budget (T)=1.75s, Average Selection Size (N*)=152026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7842 | — | |
| kNNTime Budget (T)=2.0s, Average Selection Size (N*)=172026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7763 | — | |
| SIFTTime Budget (T)=1.75s, Average Selection Size (N*)=112026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7831 | — | |
| SIFTTime Budget (T)=2.0s, Average Selection Size (N*)=132026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7656 | — |