# Language Modeling on OpenWebText 1B tokens

Source: [Language Modeling on OpenWebText 1B tokens](<https://www.sota2.com/research/sota/language-modeling-on-openwebtext-1b-tokens>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-16T09:59:02.340Z

Evaluation of language modeling performance on the OpenWebText dataset, specifically using the 1 Billion token subset, focusing on computational cost (PFLOPs) and cache efficiency metrics.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Language Modeling |
| Dataset (including any stated split) | OpenWebText 1B tokens |
| Metric | Cumulative Orthogonalization Cost (PFLOPs) |
| Unit | SIZE |
| Direction | Lower is better |
| Results loaded | 9 of 9 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Cumulative Orthogonalization Cost (PFLOPs)](<https://www.sota2.com/research/sota/language-modeling-on-openwebtext-1b-tokens.md?metric=Cumulative%20Orthogonalization%20Cost%20(PFLOPs)>): Lower is better; unit: SIZE.

- [CacheMuon Savings vs. GramMuon](<https://www.sota2.com/research/sota/language-modeling-on-openwebtext-1b-tokens.md?metric=CacheMuon%20Savings%20vs.%20GramMuon>): Higher is better; unit: PERCENT.

- [CacheMuon Savings vs. PolarExpMuon](<https://www.sota2.com/research/sota/language-modeling-on-openwebtext-1b-tokens.md?metric=CacheMuon%20Savings%20vs.%20PolarExpMuon>): Higher is better; unit: PERCENT.

- [Hit Rate](<https://www.sota2.com/research/sota/language-modeling-on-openwebtext-1b-tokens.md?metric=Hit%20Rate>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Cumulative Orthogonalization Cost (PFLOPs) | Result date | Report paper |
| --- | --- | --- | --- |
| CacheMuon | 26.67 | Not recorded | 6a30d90ad33dfb23e783d203 |
| GramMuon | 37.18 | Not recorded | 6a30d90ad33dfb23e783d203 |
| PolarExpMuon | 46.45 | Not recorded | 6a30d90ad33dfb23e783d203 |
| CacheMuon | 180.76 | Not recorded | 6a30d90ad33dfb23e783d203 |
| CacheMuon | 226.88 | Not recorded | 6a30d90ad33dfb23e783d203 |
| CacheMuon | 296.9 | Not recorded | 6a30d90ad33dfb23e783d203 |
| CacheMuon | 445.97 | Not recorded | 6a30d90ad33dfb23e783d203 |
| GramMuon | 516.16 | Not recorded | 6a30d90ad33dfb23e783d203 |
| PolarExpMuon | 644.99 | Not recorded | 6a30d90ad33dfb23e783d203 |

### CacheMuon

Cumulative Orthogonalization Cost (PFLOPs): 26.67. Result ID: 6a311e51fbc373592c2dfa15.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Small","Token count":"1B tokens"}. This record does not assert independent reproduction.

### GramMuon

Cumulative Orthogonalization Cost (PFLOPs): 37.18. Result ID: 6a311e51fbc373592c2dfa13.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Small","Token count":"1B tokens"}. This record does not assert independent reproduction.

### PolarExpMuon

Cumulative Orthogonalization Cost (PFLOPs): 46.45. Result ID: 6a311e51fbc373592c2dfa14.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa5e551e209131ce9e15f0.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Small","Token count":"1B tokens"}. This record does not assert independent reproduction.

### CacheMuon

Cumulative Orthogonalization Cost (PFLOPs): 180.76. Result ID: 6a311e51fbc373592c2dfa12.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Large","Token count":"1B tokens","gamma":"15"}. This record does not assert independent reproduction.

### CacheMuon

Cumulative Orthogonalization Cost (PFLOPs): 226.88. Result ID: 6a311e51fbc373592c2dfa11.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Large","Token count":"1B tokens","gamma":"10"}. This record does not assert independent reproduction.

### CacheMuon

Cumulative Orthogonalization Cost (PFLOPs): 296.9. Result ID: 6a311e51fbc373592c2dfa10.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Large","Token count":"1B tokens","gamma":"5"}. This record does not assert independent reproduction.

### CacheMuon

Cumulative Orthogonalization Cost (PFLOPs): 445.97. Result ID: 6a311e51fbc373592c2dfa0f.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Large","Token count":"1B tokens","gamma":"2"}. This record does not assert independent reproduction.

### GramMuon

Cumulative Orthogonalization Cost (PFLOPs): 516.16. Result ID: 6a311e51fbc373592c2dfa0d.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Large","Token count":"1B tokens"}. This record does not assert independent reproduction.

### PolarExpMuon

Cumulative Orthogonalization Cost (PFLOPs): 644.99. Result ID: 6a311e51fbc373592c2dfa0e.

[Reporting paper](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d203.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa5e551e209131ce9e15f0.md>)

Recorded evaluation configuration: {"Model":"GPT-2 Large","Token count":"1B tokens"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
