Language Modeling on C4 GPT2-125M (val)
29.65Perplexity (PPL)Adam+MoLS
Evaluation Results
| Method | Links | |
|---|---|---|
| Adam+MoLSArchitecture=GPT2-125M, Training Tokens=1.1B2026.05 | 29.65 | |
| NAdamArchitecture=GPT2-125M, Training Tokens=1.1B2026.05 | 30.26 | |
| MuonArchitecture=GPT2-125M, Training Tokens=1.1B2026.05 | 30.26 | |
| LAMBArchitecture=GPT2-125M, Training Tokens=1.1B2026.05 | 30.5 | |
| AdamArchitecture=GPT2-125M, Training Tokens=1.1B2026.05 | 31.09 |