Language Modeling on SlimPajama latest (val)
3.078Validation LossLionMuon P=2
Evaluation Results
| Method | Links | |
|---|---|---|
| LionMuon P=2Scale=124M, Architecture=LLaMA, P=22026.05 | 3.078 | |
| LionMuon P=5Scale=124M, Architecture=LLaMA, P=52026.05 | 3.08 | |
| LionMuon P=20Scale=124M, Architecture=LLaMA, P=202026.05 | 3.088 | |
| LionMuon P=1Scale=124M, Architecture=LLaMA, P=12026.05 | 3.09 | |
| SignMuon P=2Scale=124M, Architecture=LLaMA, P=22026.05 | 3.096 | |
| SignMuon P=5Scale=124M, Architecture=LLaMA, P=52026.05 | 3.099 | |
| LionMuon P=100Scale=124M, Architecture=LLaMA, P=1002026.05 | 3.109 | |
| LionMuon P=2Scale=124M, Architecture=GPT-2, P=22026.05 | 3.113 | |
| Muon (P=1)Scale=124M, Architecture=LLaMA, P=12026.05 | 3.12 | |
| LionMuon P=5Scale=124M, Architecture=GPT-2, P=52026.05 | 3.12 | |
| LionScale=124M, Architecture=LLaMA2026.05 | 3.123 | |
| SignMuon P=20Scale=124M, Architecture=LLaMA, P=202026.05 | 3.124 | |
| LionMuon P=1Scale=124M, Architecture=GPT-2, P=12026.05 | 3.125 | |
| AdamWScale=124M, Architecture=LLaMA2026.05 | 3.126 | |
| SignMuon P=2Scale=124M, Architecture=GPT-2, P=22026.05 | 3.128 | |
| SignMuon P=5Scale=124M, Architecture=GPT-2, P=52026.05 | 3.135 | |
| LionMuon P=20Scale=124M, Architecture=GPT-2, P=202026.05 | 3.137 | |
| Muon (P=1)Scale=124M, Architecture=GPT-2, P=12026.05 | 3.139 | |
| SignMuon P=100Scale=124M, Architecture=LLaMA, P=1002026.05 | 3.14 | |
| SignumScale=124M, Architecture=LLaMA2026.05 | 3.145 | |
| SignMuon P=20Scale=124M, Architecture=GPT-2, P=202026.05 | 3.163 | |
| LionMuon P=100Scale=124M, Architecture=GPT-2, P=1002026.05 | 3.165 | |
| AdamWScale=124M, Architecture=GPT-22026.05 | 3.169 | |
| LionScale=124M, Architecture=GPT-22026.05 | 3.19 | |
| SignMuon P=100Scale=124M, Architecture=GPT-2, P=1002026.05 | 3.196 | |
| SignumScale=124M, Architecture=GPT-22026.05 | 3.211 |