List your company
ResearchFormerly WizWand

Language Modeling on SlimPajama latest (val)

3.078Validation LossLionMuon P=2
3.072683.108593.14453.18041May 19, 2026
Updated 5mo ago

Evaluation results

MethodLinks
LionMuon P=2Scale=124M, Architecture=LLaMA, P=22026.053.078
LionMuon P=5Scale=124M, Architecture=LLaMA, P=52026.053.08
LionMuon P=20Scale=124M, Architecture=LLaMA, P=202026.053.088
LionMuon P=1Scale=124M, Architecture=LLaMA, P=12026.053.09
SignMuon P=2Scale=124M, Architecture=LLaMA, P=22026.053.096
SignMuon P=5Scale=124M, Architecture=LLaMA, P=52026.053.099
LionMuon P=100Scale=124M, Architecture=LLaMA, P=1002026.053.109
LionMuon P=2Scale=124M, Architecture=GPT-2, P=22026.053.113
Muon (P=1)Scale=124M, Architecture=LLaMA, P=12026.053.12
LionMuon P=5Scale=124M, Architecture=GPT-2, P=52026.053.12
LionScale=124M, Architecture=LLaMA2026.053.123
SignMuon P=20Scale=124M, Architecture=LLaMA, P=202026.053.124
LionMuon P=1Scale=124M, Architecture=GPT-2, P=12026.053.125
AdamWScale=124M, Architecture=LLaMA2026.053.126
SignMuon P=2Scale=124M, Architecture=GPT-2, P=22026.053.128
SignMuon P=5Scale=124M, Architecture=GPT-2, P=52026.053.135
LionMuon P=20Scale=124M, Architecture=GPT-2, P=202026.053.137
Muon (P=1)Scale=124M, Architecture=GPT-2, P=12026.053.139
SignMuon P=100Scale=124M, Architecture=LLaMA, P=1002026.053.14
SignumScale=124M, Architecture=LLaMA2026.053.145
SignMuon P=20Scale=124M, Architecture=GPT-2, P=202026.053.163
LionMuon P=100Scale=124M, Architecture=GPT-2, P=1002026.053.165
AdamWScale=124M, Architecture=GPT-22026.053.169
LionScale=124M, Architecture=GPT-22026.053.19
SignMuon P=100Scale=124M, Architecture=GPT-2, P=1002026.053.196
SignumScale=124M, Architecture=GPT-22026.053.211