# Language Modeling on FineWeb-Edu 300M×5B (val)

Source: [Language Modeling on FineWeb-Edu 300M×5B (val)](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-edu-300m-5b-val>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-21T12:46:24.392Z

Evaluation of language modeling performance on the validation split of the FineWeb-Edu 300M×5B subset, measured by Perplexity (PPL).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Language Modeling |
| Dataset (including any stated split) | FineWeb-Edu 300M×5B (val) |
| Metric | PPL |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [PPL](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-edu-300m-5b-val.md?metric=PPL>): Lower is better; unit: SCALAR.

## Recorded results

| Method | PPL | Result date | Report paper |
| --- | --- | --- | --- |
| FP16 reference | 23.6 | Not recorded | 69e704df7d0172140ec7f0a5 |
| FP16 reference | 23.7 | Not recorded | 69e704df7d0172140ec7f0a5 |
| FP16 reference | 23.7 | Not recorded | 69e704df7d0172140ec7f0a5 |
| QuaRot (per-Linear) W4A4 | 25.4 | Not recorded | 69e704df7d0172140ec7f0a5 |
| QuaRot (per-Linear) W4A4 | 25.4 | Not recorded | 69e704df7d0172140ec7f0a5 |
| QuaRot (per-Linear) W4A4 | 25.5 | Not recorded | 69e704df7d0172140ec7f0a5 |
| QuaRot (full) W4A4 | 26 | Not recorded | 69e704df7d0172140ec7f0a5 |
| QuaRot (full) W4A4 | 26 | Not recorded | 69e704df7d0172140ec7f0a5 |
| QuaRot (full) W4A4 | 26.1 | Not recorded | 69e704df7d0172140ec7f0a5 |
| SmoothQuant W4A4 | 39.9 | Not recorded | 69e704df7d0172140ec7f0a5 |
| SmoothQuant W4A4 | 57.6 | Not recorded | 69e704df7d0172140ec7f0a5 |
| SmoothQuant W4A4 | 58.1 | Not recorded | 69e704df7d0172140ec7f0a5 |
| Naive RTN W4A4 | 119 | Not recorded | 69e704df7d0172140ec7f0a5 |
| Naive RTN W4A4 | 671 | Not recorded | 69e704df7d0172140ec7f0a5 |
| Naive RTN W4A4 | 1727 | Not recorded | 69e704df7d0172140ec7f0a5 |

### FP16 reference

PPL: 23.6. Result ID: 69e7706247f34ce924cf1f69.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"0","Quantization":"FP16"}. This record does not assert independent reproduction.

### FP16 reference

PPL: 23.7. Result ID: 69e7706247f34ce924cf1f6a.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"FP16","Depth Registers":"true"}. This record does not assert independent reproduction.

### FP16 reference

PPL: 23.7. Result ID: 69e7706247f34ce924cf1f6b.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"FP16","Depth Registers":"true","Attention Sink":"true"}. This record does not assert independent reproduction.

### QuaRot (per-Linear) W4A4

PPL: 25.4. Result ID: 69e7706247f34ce924cf1f73.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true"}. This record does not assert independent reproduction.

### QuaRot (per-Linear) W4A4

PPL: 25.4. Result ID: 69e7706247f34ce924cf1f74.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true","Attention Sink":"true"}. This record does not assert independent reproduction.

### QuaRot (per-Linear) W4A4

PPL: 25.5. Result ID: 69e7706247f34ce924cf1f72.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"0","Quantization":"W4A4"}. This record does not assert independent reproduction.

### QuaRot (full) W4A4

PPL: 26. Result ID: 69e7706247f34ce924cf1f75.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"0","Quantization":"W4A4"}. This record does not assert independent reproduction.

### QuaRot (full) W4A4

PPL: 26. Result ID: 69e7706247f34ce924cf1f77.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true","Attention Sink":"true"}. This record does not assert independent reproduction.

### QuaRot (full) W4A4

PPL: 26.1. Result ID: 69e7706247f34ce924cf1f76.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true"}. This record does not assert independent reproduction.

### SmoothQuant W4A4

PPL: 39.9. Result ID: 69e7706247f34ce924cf1f71.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true","Attention Sink":"true"}. This record does not assert independent reproduction.

### SmoothQuant W4A4

PPL: 57.6. Result ID: 69e7706247f34ce924cf1f6f.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"0","Quantization":"W4A4"}. This record does not assert independent reproduction.

### SmoothQuant W4A4

PPL: 58.1. Result ID: 69e7706247f34ce924cf1f70.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true"}. This record does not assert independent reproduction.

### Naive RTN W4A4

PPL: 119. Result ID: 69e7706247f34ce924cf1f6e.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true","Attention Sink":"true"}. This record does not assert independent reproduction.

### Naive RTN W4A4

PPL: 671. Result ID: 69e7706247f34ce924cf1f6d.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"64","Quantization":"W4A4","Depth Registers":"true"}. This record does not assert independent reproduction.

### Naive RTN W4A4

PPL: 1727. Result ID: 69e7706247f34ce924cf1f6c.

[Reporting paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704df7d0172140ec7f0a5.md>)

Recorded evaluation configuration: {"k":"0","Quantization":"W4A4"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
