Language Modeling on FineWeb 10B (4K sequence length, val)
3.2291Validation LossLSSAR
Evaluation results
| Method | Links | |
|---|---|---|
| LSSARRe-weighting=p = 152025.01 | 3.2291 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 152025.01 | 3.6108 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 32025.01 | 4.9286 | |
| SoftmaxRe-weighting=p = 32025.01 | 5.2842 | |
| LSSARe-weighting=none2025.01 | 5.296 | |
| LSSARRe-weighting=p = 32025.01 | 5.4056 | |
| SoftmaxRe-weighting=none2025.01 | 5.4513 | |
| SSMaxRe-weighting=none2025.01 | 5.4606 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=none2025.01 | 5.5544 | |
| ReLU (Li et al., 2022)Re-weighting=none2025.01 | 5.6924 | |
| Sigmoid (Tab. A.6)Re-weighting=none2025.01 | 5.8465 | |
| SoftmaxRe-weighting=p = 152025.01 | 6.5276 | |
| Sigmoid (b = − log N)Re-weighting=none2025.01 | 8.4679 | |
| ReLU (Wortsman et al., 2023)Re-weighting=none2025.01 | 8.4982 | |
| ReLU (Shen et al., 2023)Re-weighting=none2025.01 | 8.9072 | |
| SigmoidRe-weighting=none2025.01 | 11.8355 |