Language Modeling on FineWeb 10B (1K sequence length, val)
3.1705Validation LossSSMax
Evaluation results
| Method | Links | |
|---|---|---|
| SSMaxRe-weighting=none2025.01 | 3.1705 | |
| LSSARRe-weighting=p = 32025.01 | 3.1782 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 32025.01 | 3.1841 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=none2025.01 | 3.1849 | |
| SoftmaxRe-weighting=p = 32025.01 | 3.1879 | |
| LSSARe-weighting=none2025.01 | 3.1905 | |
| LSSARRe-weighting=p = 152025.01 | 3.1905 | |
| SoftmaxRe-weighting=none2025.01 | 3.1911 | |
| Sigmoid (b = − log N)Re-weighting=none2025.01 | 3.193 | |
| SigmoidRe-weighting=none2025.01 | 3.1935 | |
| Sigmoid (Tab. A.6)Re-weighting=none2025.01 | 3.2 | |
| ReLU (Li et al., 2022)Re-weighting=none2025.01 | 3.2006 | |
| ReLU (Wortsman et al., 2023)Re-weighting=none2025.01 | 3.2143 | |
| ReLU (Shen et al., 2023)Re-weighting=none2025.01 | 3.2155 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 152025.01 | 3.2171 | |
| SoftmaxRe-weighting=p = 152025.01 | 5.3878 |