Language Modeling on LLaMA-350M pre-training (val)
2.707Validation LossALIAS Adam version
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ALIAS Adam versionweight decay=true, momentum (β)=true, cosine scheduler=true2025.06 | 2.707 | 14.984 | |
| STEEPEST DESCENTweight decay=true, momentum (β)=true, learning rate=true, cosine scheduler=true2025.06 | 2.711 | 15.044 | |
| PRODIGYweight decay=true, momentum (β)=true, cosine scheduler=true2025.06 | 2.715 | 15.105 | |
| SIGN-SGDweight decay=true, momentum (β)=true, learning rate=true, cosine scheduler=true2025.06 | 2.717 | 15.135 | |
| ADAMWweight decay=true, momentum (β)=true, learning rate=true, cosine scheduler=true2025.06 | 2.719 | 15.165 | |
| SIGN-SGDweight decay=true, learning rate=true, cosine schedule=true2025.06 | 2.819 | 16.76 | |
| ALIASweight decay=true2025.06 | 2.821 | 16.793 | |
| STEEPEST DESCENTweight decay=true, learning rate=true, cosine schedule=true2025.06 | 2.828 | 16.912 | |
| NORMALIZED SGDweight decay=true, momentum (β)=true, learning rate=true, cosine scheduler=true2025.06 | 3.46 | 31.817 | |
| NORMALIZED SGDweight decay=true, learning rate=true, cosine schedule=true2025.06 | 3.51 | 33.448 |