Language Modeling on Language Modeling Corpus (val)
9.42Average PerplexitySiameseNorm
Evaluation results
| Method | Links | |
|---|---|---|
| SiameseNormSetting=D, Learning Rate=2e-3, Training Tokens=350B2026.02 | 9.42 | |
| Hyper-Connections-2×DHCSetting=D, Learning Rate=2e-3, Training Tokens=350B, training_stability=loss spikes (*)2026.02 | 9.57 | |
| Pre-NormSetting=D, Learning Rate=2e-3, Training Tokens=350B2026.02 | 9.67 | |
| SiameseNormSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 10.43 | |
| SiameseNormSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 10.48 | |
| SiameseNormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 10.57 | |
| Hyper-Connections-2×DHCSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 10.73 | |
| Hyper-Connections-2×DHCSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 10.77 | |
| Pre-NormSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 10.84 | |
| Pre-NormSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 10.89 | |
| HybridNormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 10.91 | |
| Hyper-Connections-2×DHCSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 11.12 | |
| Pre-NormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 11.21 | |
| ResiDualSetting=B, Learning Rate=1e-3, Training Tokens=100B, training_stability=loss spikes (*)2026.02 | 11.22 | |
| Deep-NormSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 11.47 | |
| ResiDualSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 12.32 | |
| Post-NormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 12.61 | |
| Deep-NormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 12.95 | |
| ResiDualSetting=C, Learning Rate=2e-3, Training Tokens=100B, training_stability=loss spikes (*)2026.02 | 13.66 |