List your company
ResearchFormerly WizWand

Language Modeling on OpenWebText GPT-2 124M (train)

2.9552LossMVN-GradW
2.9545272.9590692.963612.968151Feb 10, 2026
Updated 8mo ago

Evaluation results

MethodLinks
MVN-GradWBackbone=GPT-2 124M, Block Size=1024, Effective Batch Size=480, Precision=bf16, Weight Decay=Decoupled (0.1)2026.022.9552
AdaBeliefWBackbone=GPT-2 124M, Block Size=1024, Effective Batch Size=480, Precision=bf16, Weight Decay=Decoupled (0.1)2026.022.9558
AdamWBackbone=GPT-2 124M, Block Size=1024, Effective Batch Size=480, Precision=bf16, Weight Decay=Decoupled (0.1)2026.022.9715
LaPropWBackbone=GPT-2 124M, Block Size=1024, Effective Batch Size=480, Precision=bf16, Weight Decay=Decoupled (0.1)2026.022.972