# Language Modeling on FineWeb 10B (4K sequence length, val) Source: [Language Modeling on FineWeb 10B (4K sequence length, val)]() Collection index: [llms.txt]() Data updated: 2026-06-02T09:56:30.193Z Evaluation of language model performance on the FineWeb 10 Billion token dataset, specifically restricted to sequences of 4,000 tokens, measured by Validation Loss. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Language Modeling | | Dataset (including any stated split) | FineWeb 10B 4K sequence length | | Metric | Validation Loss | | Unit | SCALAR | | Direction | Lower is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Validation Loss](): Lower is better; unit: SCALAR. ## Recorded results | Method | Validation Loss | Result date | Report paper | | --- | --- | --- | --- | | LSSAR | 3.2291 | Not recorded | 68fa7a8b1e209131cea83d15 | | Sigmoid (b = − log N, l1-norm) | 3.6108 | Not recorded | 68fa7a8b1e209131cea83d15 | | Sigmoid (b = − log N, l1-norm) | 4.9286 | Not recorded | 68fa7a8b1e209131cea83d15 | | Softmax | 5.2842 | Not recorded | 68fa7a8b1e209131cea83d15 | | LSSA | 5.296 | Not recorded | 68fa7a8b1e209131cea83d15 | | LSSAR | 5.4056 | Not recorded | 68fa7a8b1e209131cea83d15 | | Softmax | 5.4513 | Not recorded | 68fa7a8b1e209131cea83d15 | | SSMax | 5.4606 | Not recorded | 68fa7a8b1e209131cea83d15 | | Sigmoid (b = − log N, l1-norm) | 5.5544 | Not recorded | 68fa7a8b1e209131cea83d15 | | ReLU (Li et al., 2022) | 5.6924 | Not recorded | 68fa7a8b1e209131cea83d15 | | Sigmoid (Tab. A.6) | 5.8465 | Not recorded | 68fa7a8b1e209131cea83d15 | | Softmax | 6.5276 | Not recorded | 68fa7a8b1e209131cea83d15 | | Sigmoid (b = − log N) | 8.4679 | Not recorded | 68fa7a8b1e209131cea83d15 | | ReLU (Wortsman et al., 2023) | 8.4982 | Not recorded | 68fa7a8b1e209131cea83d15 | | ReLU (Shen et al., 2023) | 8.9072 | Not recorded | 68fa7a8b1e209131cea83d15 | | Sigmoid | 11.8355 | Not recorded | 68fa7a8b1e209131cea83d15 | ### LSSAR Validation Loss: 3.2291. Result ID: 6a1ea84afbc373592c19448f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"p = 15"}. This record does not assert independent reproduction. ### Sigmoid (b = − log N, l1-norm) Validation Loss: 3.6108. Result ID: 6a1ea84afbc373592c19448e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"p = 15"}. This record does not assert independent reproduction. ### Sigmoid (b = − log N, l1-norm) Validation Loss: 4.9286. Result ID: 6a1ea84afbc373592c19448b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"p = 3"}. This record does not assert independent reproduction. ### Softmax Validation Loss: 5.2842. Result ID: 6a1ea84afbc373592c19448a. [Reporting paper]() Recorded evaluation configuration: {"Re-weighting":"p = 3"}. This record does not assert independent reproduction. ### LSSA Validation Loss: 5.296. Result ID: 6a1ea84afbc373592c194489. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### LSSAR Validation Loss: 5.4056. Result ID: 6a1ea84afbc373592c19448c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"p = 3"}. This record does not assert independent reproduction. ### Softmax Validation Loss: 5.4513. Result ID: 6a1ea84afbc373592c194480. [Reporting paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### SSMax Validation Loss: 5.4606. Result ID: 6a1ea84afbc373592c194481. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### Sigmoid (b = − log N, l1-norm) Validation Loss: 5.5544. Result ID: 6a1ea84afbc373592c194484. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### ReLU (Li et al., 2022) Validation Loss: 5.6924. Result ID: 6a1ea84afbc373592c194487. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### Sigmoid (Tab. A.6) Validation Loss: 5.8465. Result ID: 6a1ea84afbc373592c194485. [Reporting paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### Softmax Validation Loss: 6.5276. Result ID: 6a1ea84afbc373592c19448d. [Reporting paper]() Recorded evaluation configuration: {"Re-weighting":"p = 15"}. This record does not assert independent reproduction. ### Sigmoid (b = − log N) Validation Loss: 8.4679. Result ID: 6a1ea84afbc373592c194483. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### ReLU (Wortsman et al., 2023) Validation Loss: 8.4982. Result ID: 6a1ea84afbc373592c194486. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### ReLU (Shen et al., 2023) Validation Loss: 8.9072. Result ID: 6a1ea84afbc373592c194488. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. ### Sigmoid Validation Loss: 11.8355. Result ID: 6a1ea84afbc373592c194482. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Re-weighting":"none"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.