Language Modeling on FineWeb 10B (8K sequence length)
3.3171Validation LossLSSAR
Evaluation results
| Method | Links | |
|---|---|---|
| LSSARRe-weighting=p = 152025.01 | 3.3171 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 152025.01 | 3.8587 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 32025.01 | 5.745 | |
| LSSARe-weighting=none2025.01 | 5.9403 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=none2025.01 | 6.1846 | |
| SoftmaxRe-weighting=p = 32025.01 | 6.2339 | |
| SoftmaxRe-weighting=none2025.01 | 6.2823 | |
| LSSARRe-weighting=p = 32025.01 | 6.3007 | |
| SSMaxRe-weighting=none2025.01 | 6.4384 | |
| ReLU (Li et al., 2022)Re-weighting=none2025.01 | 6.4561 | |
| Sigmoid (Tab. A.6)Re-weighting=none2025.01 | 6.5701 | |
| SoftmaxRe-weighting=p = 152025.01 | 7.0183 | |
| Sigmoid (b = − log N)Re-weighting=none2025.01 | 9.5939 | |
| ReLU (Wortsman et al., 2023)Re-weighting=none2025.01 | 10.346 | |
| ReLU (Shen et al., 2023)Re-weighting=none2025.01 | 10.7266 | |
| SigmoidRe-weighting=none2025.01 | 14.4995 |