Language Modeling on OpenWebText (train)
2.5243Train LossMidpoint
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MidpointModel Size=GPT Large, Number of Parameters=772M2025.11 | 2.5243 | — | |
| LeapfrogModel Size=GPT Large, Number of Parameters=772M2025.11 | 2.575 | — | |
| BaselineModel Size=GPT Large, Number of Parameters=772M2025.11 | 2.6544 | — | |
| ADAMSsynchronization=DENSE all-reduce2026.07 | 2.77 | — | |
| SCAPEd=0.12026.07 | 2.77 | — | |
| ADAMWsynchronization=DENSE all-reduce2026.07 | 2.8 | — | |
| SCAPEd=0.012026.07 | 2.81 | — | |
| MidpointModel Size=GPT Small, Number of Parameters=124M2025.11 | 2.8681 | — | |
| BaselineModel Size=GPT Small, Number of Parameters=124M2025.11 | 2.881 | — | |
| GPTEmbedding Size=768, Number of Parameters=124M, Number of Runs=52025.12 | 2.905 | 0.006 | |
| LeapfrogModel Size=GPT Small, Number of Parameters=124M2025.11 | 2.9245 | — | |
| sHCModel Scale=L2026.03 | 2.998 | — | |
| mHCModel Scale=L2026.03 | 3 | — | |
| mHC-liteModel Scale=L2026.03 | 3.001 | — | |
| RCModel Scale=L2026.03 | 3.052 | — | |
| HCModel Scale=L2026.03 | 3.111 | — | |
| GPT2-81M-LOOPParameters=81M, Layers=6, Loops=62024.09 | 3.13 | — | |
| GPT2-67M-LOOPParameters=67M, Layers=4, Loops=122024.09 | 3.15 | — | |
| sHCModel Scale=M2026.03 | 3.225 | — | |
| mHCModel Scale=M2026.03 | 3.268 | — | |
| mHC-liteModel Scale=M2026.03 | 3.271 | — | |
| HCModel Scale=M2026.03 | 3.282 | — | |
| RCModel Scale=M2026.03 | 3.347 | — | |
| GPT_Rec_parallelEmbedding Size=768, Number of Parameters=85M, Number of Runs=52025.12 | 3.447 | 0.046 | |
| NRGPT_H_FF2WEmbedding Size=768, Number of Parameters=90M, Number of Runs=32025.12 | 3.456 | 0.076 |