Language Modeling on GPT-2 Pretraining Data (train)
2.9167Training LossNAMO-D
Evaluation results
| Method | Links | ||
|---|---|---|---|
| NAMO-DModel=GPT-2 (124M), Steps=50K2026.02 | 2.9167 | — | |
| NAMOModel=GPT-2 (124M), Steps=50K2026.02 | 2.9272 | — | |
| NAMO-DModel=GPT-2 (355M), Steps=10K2026.02 | 2.9351 | — | |
| NAMOModel=GPT-2 (355M), Steps=10K2026.02 | 2.9359 | — | |
| MuonModel=GPT-2 (355M), Steps=10K2026.02 | 2.9524 | — | |
| AdamWModel=GPT-2 (355M), Steps=10K2026.02 | 2.976 | — | |
| MuonModel=GPT-2 (124M), Steps=50K2026.02 | 3.0265 | — | |
| AdamWModel=GPT-2 (124M), Steps=50K2026.02 | 3.0456 | — | |
| StoSignSGDv2Precision=FP82026.04 | 3.394 | 29.78 | |
| StoSignSGDPrecision=FP82026.04 | 3.41 | 30.26 | |
| LionPrecision=FP82026.04 | 3.452 | 31.55 | |
| SignSGDPrecision=FP82026.04 | 3.539 | 34.43 |