List your company
ResearchFormerly WizWand

Language Modeling on GPT-2 Pre-training (val)

2.493Validation LossDense
2.469682.627092.78452.94191Sep 13, 2024Dec 9, 2024Mar 6, 2025Jun 2, 2025Aug 28, 2025Nov 23, 2025Feb 19, 2026
Updated 8mo ago

Evaluation results

MethodLinks
DenseParams=774M, Pre-train=Dense2024.092.493
S-STEParams=774M, Pre-train=S-STE2024.092.547
T-SR-STE+DFParams=774M, Pre-train=T-SR-STE+DF [20]2024.092.564
DenseParams=350M, Pre-train=Dense2024.092.618
T-SR-STE+DFParams=350M, Pre-train=T-SR-STE+DF [20]2024.092.688
SR-STEParams=350M, Pre-train=SR-STE2024.092.69
S-STEParams=350M, Pre-train=S-STE2024.092.713
T-SR-STEParams=350M, Pre-train=T-SR-STE2024.092.718
DenseParams=124M, Pre-train=Dense2024.092.907
NAMO-DModel=GPT-2 (355M), Steps=10K2026.022.9507
NAMOModel=GPT-2 (355M), Steps=10K2026.022.9516
T-SR-STE+DFParams=124M, Pre-train=T-SR-STE+DF [20]2024.092.952
MuonModel=GPT-2 (355M), Steps=10K2026.022.9684
SR-STEParams=124M, Pre-train=SR-STE2024.092.982
S-STEParams=124M, Pre-train=S-STE2024.092.984
AdamWModel=GPT-2 (355M), Steps=10K2026.022.9914
NAMO-DModel=GPT-2 (124M), Steps=50K2026.023.0246
NAMOModel=GPT-2 (124M), Steps=50K2026.023.0351
MuonModel=GPT-2 (124M), Steps=50K2026.023.0435
AdamWModel=GPT-2 (124M), Steps=50K2026.023.0643
T-SR-STEParams=124M, Pre-train=T-SR-STE2024.093.076