Zero-shot Reasoning on Downstream Task Suite (ARC-E, ARC-C, HS, OBQA, PIQA, WG, Arith.)
78.1ARC-EPre-Norm
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Pre-NormSetting=D, Learning Rate=2e-3, Training Tokens=350B2026.02 | 78.1 | 42.8 | 63.5 | 41.6 | 76 | 62 | 36.2 | 57.17 | |
| Hyper-Connections-2×DHCSetting=D, Learning Rate=2e-3, Training Tokens=350B2026.02 | 76.3 | 43.1 | 63.5 | 42.4 | 75.3 | 61.3 | 33.6 | 56.5 | |
| SiameseNormSetting=D, Learning Rate=2e-3, Training Tokens=350B2026.02 | 75.6 | 44.1 | 64.7 | 44.6 | 76 | 62.5 | 43.4 | 58.7 | |
| Hyper-Connections-2×DHCSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 73.7 | 40.1 | 58.2 | 40 | 73.8 | 60.4 | 30.6 | 53.83 | |
| SiameseNormSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 73.5 | 38.5 | 59.6 | 41.4 | 74.6 | 62.2 | 39.6 | 55.63 | |
| Hyper-Connections-2×DHCSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 72.6 | 39.1 | 57.7 | 41.2 | 73.8 | 59.9 | 27.9 | 53.17 | |
| SiameseNormSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 72.5 | 39.8 | 59 | 41 | 74 | 59 | 29.4 | 53.53 | |
| HybridNormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 72.3 | 38.8 | 56.4 | 38.4 | 74.2 | 57 | 27.6 | 52.1 | |
| SiameseNormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 72.1 | 36.8 | 57.8 | 40.6 | 72.5 | 57.6 | 28.4 | 52.26 | |
| Pre-NormSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 71.9 | 37.8 | 56.4 | 39.8 | 73.8 | 56.7 | 27 | 51.91 | |
| Pre-NormSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 71.6 | 40.8 | 57.7 | 41.4 | 73.7 | 57.3 | 28.1 | 52.94 | |
| Hyper-Connections-2×DHCSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 70.9 | 38.8 | 54.6 | 39 | 72.6 | 55.2 | 26.9 | 51.14 | |
| Pre-NormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 70.2 | 37.5 | 54.1 | 36.2 | 73.3 | 56.5 | 27.7 | 50.79 | |
| ResiDualSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 70 | 39.5 | 55.1 | 41.2 | 72.8 | 57 | 27.5 | 51.87 | |
| Deep-NormSetting=B, Learning Rate=1e-3, Training Tokens=100B2026.02 | 68.4 | 35.4 | 53.7 | 38.8 | 72.1 | 56.2 | 27.7 | 50.33 | |
| Post-NormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 67.5 | 34.1 | 48 | 36.6 | 70.6 | 53.6 | 27.1 | 48.21 | |
| ResiDualSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 66.8 | 33.4 | 49.6 | 36.8 | 71.1 | 54.5 | 25.8 | 48.29 | |
| Deep-NormSetting=A, Learning Rate=4e-4, Training Tokens=100B2026.02 | 64.9 | 30.1 | 46.5 | 36.4 | 69.2 | 55.7 | 26.3 | 47.01 | |
| ResiDualSetting=C, Learning Rate=2e-3, Training Tokens=100B2026.02 | 64.4 | 33.8 | 45.2 | 35.4 | 69 | 53.6 | 26.2 | 46.8 |