Language Modeling on FineWeb 10B (2K sequence length)
3.193Validation LossLSSAR
Evaluation results
| Method | Links | |
|---|---|---|
| LSSARRe-weighting=p = 152025.01 | 3.193 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 152025.01 | 3.3499 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=p = 32025.01 | 3.7387 | |
| SoftmaxRe-weighting=p = 32025.01 | 4.0277 | |
| SSMaxRe-weighting=none2025.01 | 4.0915 | |
| LSSARe-weighting=none2025.01 | 4.1301 | |
| SoftmaxRe-weighting=none2025.01 | 4.1662 | |
| LSSARRe-weighting=p = 32025.01 | 4.2383 | |
| Sigmoid (b = − log N, l1-norm)Re-weighting=none2025.01 | 4.347 | |
| Sigmoid (Tab. A.6)Re-weighting=none2025.01 | 4.3811 | |
| ReLU (Li et al., 2022)Re-weighting=none2025.01 | 4.5192 | |
| SoftmaxRe-weighting=p = 152025.01 | 5.9491 | |
| ReLU (Wortsman et al., 2023)Re-weighting=none2025.01 | 6.2662 | |
| ReLU (Shen et al., 2023)Re-weighting=none2025.01 | 6.5573 | |
| Sigmoid (b = − log N)Re-weighting=none2025.01 | 6.583 | |
| SigmoidRe-weighting=none2025.01 | 7.4554 |