Language Modeling on Pretraining Dataset
2.1506Train Loss (PT)LLR
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LLRModel Architecture=LLaMa-1B, Training Tokens=100B2026.05 | 2.1506 | — | 8.74 | |
| UniformModel Architecture=LLaMa-1B, Training Tokens=100B2026.05 | 2.1543 | — | 8.66 | |
| LLRModel Architecture=LLaMa-3B, Training Tokens=30B2026.05 | 2.1722 | — | 8.86 | |
| UniformModel Architecture=LLaMa-3B, Training Tokens=30B2026.05 | 2.1758 | — | 9.02 | |
| BHyTBackbone=Llama-3B2025.12 | 3.133 | 3.107 | 22.346 | |
| LNSBackbone=Llama-3B2025.12 | 3.16 | 3.139 | 23.091 | |
| Peri-LNBackbone=Llama-3B2025.12 | 3.165 | 3.142 | 23.156 | |
| RMSNormBackbone=Llama-3B2025.12 | 3.203 | 3.18 | 24.04 | |
| BHyTModel=Llama-1B, Evaluation Protocol=Pretraining-only2025.12 | 3.268 | 3.254 | 25.908 | |
| LNSModel=Llama-1B, Evaluation Protocol=Pretraining-only2025.12 | 3.28 | 3.271 | 26.342 | |
| RMSNormModel=Llama-1B, Evaluation Protocol=Pretraining-only2025.12 | 3.281 | 3.272 | 26.353 | |
| Peri-LNModel=Llama-1B, Evaluation Protocol=Pretraining-only2025.12 | 3.288 | 3.279 | 26.545 | |
| DyTModel=Llama-1B, Evaluation Protocol=Pretraining-only2025.12 | 3.709 | 3.696 | 40.294 | |
| DyTBackbone=Llama-3B2025.12 | 3.877 | 3.855 | 47.244 |