List your company
ResearchFormerly WizWand

Language Model Pre-training on C4 Llama 2 pre-training (val)

13.19PerplexityRMSProp+Magma
12.414817.647422.8828.1126Feb 17, 2026
Updated 8mo ago

Evaluation results

MethodLinks
RMSProp+MagmaModel Scale=1B2026.0213.19
Adam+MagmaModel Scale=1B2026.0213.71
LaProp+MagmaModel Scale=1B2026.0213.82
APOLLO+SGGModel Scale=1B2026.0213.95
APOLLOModel Scale=1B2026.0214.2
Adam+SGGModel Scale=1B2026.0214.3
MuonModel Scale=1B2026.0214.52
AdafactorModel Scale=1B2026.0215.19
C-AdamModel Scale=1B2026.0215.92
RMSProp+MagmaModel Scale=350M2026.0216.16
AdamModel Scale=1B2026.0216.35
LaProp+MagmaModel Scale=350M2026.0216.37
LaPropModel Scale=1B2026.0216.38
Adam+MagmaModel Scale=350M2026.0216.41
APOLLO+SGGModel Scale=350M2026.0216.54
APOLLOModel Scale=350M2026.0216.85
MuonModel Scale=350M2026.0217.09
Adam+SGGModel Scale=350M2026.0217.28
RMSPropModel Scale=350M2026.0217.47
AdafactorModel Scale=350M2026.0217.74
AdamModel Scale=350M2026.0218.42
LaPropModel Scale=350M2026.0218.56
C-AdamModel Scale=350M2026.0218.58
RMSProp+MagmaModel Scale=130M2026.0221.66
Adam+MagmaModel Scale=130M2026.0222.08
LaProp+MagmaModel Scale=130M2026.0222.16
Adam+SGGModel Scale=130M2026.0222.18
MuonModel Scale=130M2026.0222.34
APOLLO+SGGModel Scale=130M2026.0222.52
RMSPropModel Scale=130M2026.0222.64
APOLLOModel Scale=130M2026.0222.94
LaPropModel Scale=130M2026.0223.07
C-AdamModel Scale=130M2026.0223.59
AdafactorModel Scale=130M2026.0223.98
AdamModel Scale=130M2026.0224.77
RMSProp+MagmaModel Scale=60M2026.0228.55
MuonModel Scale=60M2026.0228.93
LaProp+MagmaModel Scale=60M2026.0229.05
Adam+MagmaModel Scale=60M2026.0229.09
RMSPropModel Scale=60M2026.0229.29
C-AdamModel Scale=60M2026.0229.7
LaPropModel Scale=60M2026.0229.98
APOLLO+SGGModel Scale=60M2026.0230.18
Adam+SGGModel Scale=60M2026.0230.31
AdamModel Scale=60M2026.0230.79
APOLLOModel Scale=60M2026.0231.55
AdafactorModel Scale=60M2026.0232.57