# Language Modeling on FineWeb (val)

Source: [Language Modeling on FineWeb (val)](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-30T10:42:01.739Z

Evaluation of language model performance on the validation split of the FineWeb dataset, typically measured using perplexity.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Language Modeling |
| Dataset (including any stated split) | FineWeb (val) |
| Metric | Validation Loss |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 363 of 363 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Validation Loss](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Validation%20Loss>): Lower is better; unit: SCALAR.

- [Perplexity](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Perplexity>): Lower is better; unit: SCALAR.

- [BPB](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=BPB>): Lower is better; unit: SCALAR.

- [Steps to Target](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Steps%20to%20Target>): Lower is better; unit: SCALAR.

- [Step Saving (%)](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Step%20Saving%20(%25)>): Higher is better; unit: PERCENT.

- [Per-Step Time (ms)](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Per-Step%20Time%20(ms)>): Lower is better; unit: TIME.

- [Relative Loss Increase](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Relative%20Loss%20Increase>): Lower is better; unit: PERCENT.

## Recorded results

| Method | Validation Loss | Result date | Report paper |
| --- | --- | --- | --- |
| Muon (Sync) | 1.906 | Not recorded | 6a434da8ba73165dbb743f9a |
| Muon (Async + EF) | 1.906 | Not recorded | 6a434da8ba73165dbb743f9a |
| Muon (Async) | 1.911 | Not recorded | 6a434da8ba73165dbb743f9a |
| UMTAM | 2.03 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.031 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.031 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.044 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.044 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.045 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.045 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.049 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.049 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.05 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.05 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.058 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.059 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.059 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.059 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.08 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.081 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.081 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.081 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.107 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.107 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.109 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.109 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.11 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.111 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.112 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.112 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.112 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.113 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.113 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.116 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.116 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.118 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.118 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.12 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.121 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.122 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.123 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.124 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.125 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.126 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.127 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.128 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.129 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.129 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.13 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.13 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.131 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.131 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.131 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.131 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.131 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.131 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.132 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.133 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.133 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.134 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.134 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.135 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.135 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.135 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.136 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.136 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.138 | Not recorded | 6948d08b423ea42265729fd9 |
| MoFaSGD | 2.138 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.139 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.139 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.139 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.14 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.143 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.143 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.144 | Not recorded | 6948d08b423ea42265729fd9 |
| GaLore | 2.145 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.145 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.145 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.146 | Not recorded | 6948d08b423ea42265729fd9 |
| UMTAM | 2.151 | Not recorded | 6948d08b423ea42265729fd9 |

### Muon (Sync)

Validation Loss: 1.906. Result ID: 6a439d5efbc373592c96da09.

[Reporting paper](<https://www.sota2.com/research/paper/6a434da8ba73165dbb743f9a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a434da8ba73165dbb743f9a.md>)

Recorded evaluation configuration: {"Optimizer":"Muon","Parallelism Strategy":"Sync","Model Scale":"10B MoE","Training Horizon":"200B tokens"}. This record does not assert independent reproduction.

### Muon (Async + EF)

Validation Loss: 1.906. Result ID: 6a439d5efbc373592c96da0b.

[Reporting paper](<https://www.sota2.com/research/paper/6a434da8ba73165dbb743f9a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a434da8ba73165dbb743f9a.md>)

Recorded evaluation configuration: {"Optimizer":"Muon","Parallelism Strategy":"Async + EF","Model Scale":"10B MoE","Training Horizon":"200B tokens"}. This record does not assert independent reproduction.

### Muon (Async)

Validation Loss: 1.911. Result ID: 6a439d5efbc373592c96da0a.

[Reporting paper](<https://www.sota2.com/research/paper/6a434da8ba73165dbb743f9a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a434da8ba73165dbb743f9a.md>)

Recorded evaluation configuration: {"Optimizer":"Muon","Parallelism Strategy":"Async","Model Scale":"10B MoE","Training Horizon":"200B tokens"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.03. Result ID: 699950b35698665ec4bb4d68.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"learning rate (η)":"1e-4","rank (r)":32,"step":1000,"model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.031. Result ID: 699950b35698665ec4bb4d67.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"learning rate (η)":"1e-4","rank (r)":16,"step":1000,"model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.031. Result ID: 699950b35698665ec4bb4d69.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"learning rate (η)":"1e-4","rank (r)":128,"step":1000,"model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.044. Result ID: 699950b35698665ec4bb4d13.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.044. Result ID: 699950b35698665ec4bb4d15.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.045. Result ID: 699950b35698665ec4bb4d12.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.045. Result ID: 699950b35698665ec4bb4d14.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.049. Result ID: 699950b35698665ec4bb4d06.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":800,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.049. Result ID: 699950b35698665ec4bb4d07.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":800,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.05. Result ID: 699950b35698665ec4bb4d08.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":800,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.05. Result ID: 699950b35698665ec4bb4d09.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":800,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.058. Result ID: 699950b35698665ec4bb4cfa.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":600,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.059. Result ID: 699950b35698665ec4bb4cfb.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":600,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.059. Result ID: 699950b35698665ec4bb4cfc.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":600,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.059. Result ID: 699950b35698665ec4bb4cfd.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":600,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.08. Result ID: 699950b35698665ec4bb4cf1.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":400,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.081. Result ID: 699950b35698665ec4bb4cee.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":400,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.081. Result ID: 699950b35698665ec4bb4cef.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":400,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.081. Result ID: 699950b35698665ec4bb4cf0.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":400,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.107. Result ID: 699950b35698665ec4bb4d49.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.107. Result ID: 699950b35698665ec4bb4d4d.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.109. Result ID: 699950b35698665ec4bb4d48.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.109. Result ID: 699950b35698665ec4bb4d4c.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.11. Result ID: 699950b35698665ec4bb4d45.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.111. Result ID: 699950b35698665ec4bb4d47.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.112. Result ID: 699950b35698665ec4bb4d43.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.112. Result ID: 699950b35698665ec4bb4d44.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.112. Result ID: 699950b35698665ec4bb4d4b.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.113. Result ID: 699950b35698665ec4bb4d46.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.113. Result ID: 699950b35698665ec4bb4d4a.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.116. Result ID: 699950b35698665ec4bb4d41.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.116. Result ID: 699950b35698665ec4bb4d42.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.118. Result ID: 699950b35698665ec4bb4d1d.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.118. Result ID: 699950b35698665ec4bb4d40.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.12. Result ID: 699950b35698665ec4bb4d3f.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.121. Result ID: 699950b35698665ec4bb4d11.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":800,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.122. Result ID: 699950b35698665ec4bb4d1c.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.123. Result ID: 699950b35698665ec4bb4d3e.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.124. Result ID: 699950b35698665ec4bb4d1b.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.125. Result ID: 699950b35698665ec4bb4d10.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":800,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.126. Result ID: 699950b35698665ec4bb4d05.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":600,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.127. Result ID: 699950b35698665ec4bb4d1a.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":1000,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.128. Result ID: 699950b35698665ec4bb4d0f.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":800,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.129. Result ID: 699950b35698665ec4bb4d3d.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"200","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.129. Result ID: 699950b35698665ec4bb4d6a.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"learning rate (η)":"1e-4","rank (r)":8,"step":1000,"model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.13. Result ID: 699950b35698665ec4bb4ce3.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":200,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.13. Result ID: 699950b35698665ec4bb4ce4.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":200,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.131. Result ID: 699950b35698665ec4bb4ce2.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":200,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.131. Result ID: 699950b35698665ec4bb4ce5.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":200,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.131. Result ID: 699950b35698665ec4bb4d0e.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":800,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.131. Result ID: 699950b35698665ec4bb4d33.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.131. Result ID: 699950b35698665ec4bb4d34.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.131. Result ID: 699950b35698665ec4bb4d3c.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"200","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.132. Result ID: 699950b35698665ec4bb4d32.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.133. Result ID: 699950b35698665ec4bb4d04.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":600,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.133. Result ID: 699950b35698665ec4bb4d35.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"1000","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.134. Result ID: 699950b35698665ec4bb4d03.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":600,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.134. Result ID: 699950b35698665ec4bb4d31.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.135. Result ID: 699950b35698665ec4bb4d2e.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.135. Result ID: 699950b35698665ec4bb4d30.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.135. Result ID: 699950b35698665ec4bb4d3b.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"200","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.136. Result ID: 699950b35698665ec4bb4cf9.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":400,"Rank":128,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.136. Result ID: 699950b35698665ec4bb4d2f.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"800","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.138. Result ID: 699950b35698665ec4bb4d2a.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### MoFaSGD

Validation Loss: 2.138. Result ID: 699950b35698665ec4bb4d3a.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79fa1e209131cea8066e.md>)

Recorded evaluation configuration: {"Step":"200","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.139. Result ID: 699950b35698665ec4bb4d02.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":600,"Rank":8,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.139. Result ID: 699950b35698665ec4bb4d2c.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.139. Result ID: 699950b35698665ec4bb4d2d.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.14. Result ID: 699950b35698665ec4bb4d2b.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"600","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.143. Result ID: 699950b35698665ec4bb4cf8.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":400,"Rank":32,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.143. Result ID: 699950b35698665ec4bb4d66.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"learning rate (η)":"1e-4","rank (r)":8,"step":1000,"model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.144. Result ID: 699950b35698665ec4bb4d28.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"32","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### GaLore

Validation Loss: 2.145. Result ID: 699950b35698665ec4bb4cf7.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73661e209131cea5c38a.md>)

Recorded evaluation configuration: {"Step":400,"Rank":16,"Learning rate":"3e-5","Model":"GPT-2 (124M)"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.145. Result ID: 699950b35698665ec4bb4d26.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"8","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.145. Result ID: 699950b35698665ec4bb4d29.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"128","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.146. Result ID: 699950b35698665ec4bb4d27.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"400","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

### UMTAM

Validation Loss: 2.151. Result ID: 699950b35698665ec4bb4d23.

[Reporting paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6948d08b423ea42265729fd9.md>)

Recorded evaluation configuration: {"Step":"200","Rank (r)":"16","Learning rate (η)":"5e-4"}. This record does not assert independent reproduction.

Page 1 of 4. This is an index page, not the full collection.

- [Next index page](<https://www.sota2.com/research/sota/language-modeling-on-fineweb-val.md?metric=Validation%20Loss&page=2>): Follow this link to discover more pages.
