Language Modeling on Pre-training corpus (train)
15.71PerplexityPre-LN + LayerNorm Scaling
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Pre-LN + LayerNorm ScalingModel Size=LLaMA-1B, Training Tokens=8.9B2025.02 | 15.71 | — | |
| Pre-LNModel Size=LLaMA-1B, Training Tokens=8.9B2025.02 | 17.02 | — | |
| Pre-LN + LayerNorm ScalingModel Size=LLaMA-350M, Training Tokens=6.0B2025.02 | 18.2 | — | |
| Pre-LNModel Size=LLaMA-350M, Training Tokens=6.0B2025.02 | 19.58 | — | |
| Pre-LN + LayerNorm ScalingModel Size=LLaMA-250M, Training Tokens=3.9B2025.02 | 20.35 | — | |
| Mix-LNModel Size=LLaMA-250M, Training Tokens=3.9B2025.02 | 21.39 | — | |
| Pre-LNModel Size=LLaMA-250M, Training Tokens=3.9B2025.02 | 21.92 | — | |
| DeepNormModel Size=LLaMA-250M, Training Tokens=3.9B2025.02 | 22.77 | — | |
| Pre-LN + LayerNorm ScalingModel Size=LLaMA-130M, Training Tokens=2.2B2025.02 | 25.76 | — | |
| Mix-LNModel Size=LLaMA-130M, Training Tokens=2.2B2025.02 | 26.07 | — | |
| Pre-LNModel Size=LLaMA-130M, Training Tokens=2.2B2025.02 | 26.73 | — | |
| Post-LNModel Size=LLaMA-130M, Training Tokens=2.2B2025.02 | 26.95 | — | |
| DeepNormModel Size=LLaMA-130M, Training Tokens=2.2B2025.02 | 27.17 | — | |
| DeepNormModel Size=LLaMA-350M, Training Tokens=6.0B2025.02 | 1,362.59 | — | |
| Mix-LNModel Size=LLaMA-350M, Training Tokens=6.0B2025.02 | 1,363.21 | — | |
| Post-LNModel Size=LLaMA-350M, Training Tokens=6.0B2025.02 | 1,368.33 | — | |
| Post-LNModel Size=LLaMA-1B, Training Tokens=8.9B2025.02 | 1,390.75 | — | |
| DeepNormModel Size=LLaMA-1B, Training Tokens=8.9B2025.02 | 1,409.08 | — | |
| Post-LNModel Size=LLaMA-250M, Training Tokens=3.9B2025.02 | 1,409.79 | — | |
| Mix-LNModel Size=LLaMA-1B, Training Tokens=8.9B2025.02 | 1,414.78 | — | |
| BMoJo (Fading + Eidetic)Model Scale=370M, Context Window=512 tokens2024.07 | — | 2.67 | |
| BMoJo (Fading + Eidetic)Model Scale=1.4B, Context Window=512 tokens2024.07 | — | 2.26 | |
| BMoJo (Fading)Model Scale=370M, Context Window=512 tokens2024.07 | — | 2.68 | |
| BMoJo (Fading)Model Scale=1.4B, Context Window=512 tokens2024.07 | — | 2.27 | |
| Hybrid (Sliding Attention + SSM)Model Scale=370M2024.07 | — | 2.69 | |
| Hybrid (Sliding Attention + SSM)Model Scale=1.4B2024.07 | — | 2.42 | |
| Mamba (SSM)Model Scale=370M2024.07 | — | 2.62 | |
| Mamba (SSM)Model Scale=1.4B2024.07 | — | 2.37 | |
| Mistral (Full-Attention)Model Scale=370M, Context Window=1024 tokens2024.07 | — | 2.56 | |
| Mistral (Full-Attention)Model Scale=1.4B, Context Window=1024 tokens2024.07 | — | 2.27 |