List your company
ResearchFormerly WizWand

Language Modeling on NanoGPT OpenWebText

391,100Throughput (tokens/s)AdamW
-148101,426203,000304,574Mar 18, 2026
Updated 7mo ago

Evaluation results

MethodLinks
AdamWGPU=GH200, (B, T)=(12, 2048), Model size=GPT-2 small2026.03391,100—1.82
AdamWGPU=GH200, (B, T)=(8, 1024), Model size=GPT-2 small2026.03342,400—3.14
MUD1GPU=GH200, (B, T)=(12, 2048), Model size=GPT-2 small2026.03295,3001.38—
MuonGPU=GH200, (B, T)=(12, 2048), Model size=GPT-2 small2026.03214,600——
MUD1GPU=GH200, (B, T)=(8, 1024), Model size=GPT-2 small2026.03185,8001.7—
AdamWGPU=A100, (B, T)=(8, 1024), Model size=GPT-2 small2026.03131,800—2.81
AdamWGPU=A100, (B, T)=(12, 2048), Model size=GPT-2 small2026.03130,000—1.62
AdamWGPU=A100, (B, T)=(8, 4096), Model size=GPT-2 small2026.03125,700—1.43
MUD1GPU=A100, (B, T)=(8, 4096), Model size=GPT-2 small2026.03113,7001.29—
MuonGPU=GH200, (B, T)=(8, 1024), Model size=GPT-2 small2026.03109,000——
MUD1GPU=A100, (B, T)=(12, 2048), Model size=GPT-2 small2026.03108,0001.35—
MuonGPU=A100, (B, T)=(8, 4096), Model size=GPT-2 small2026.0388,000——
MUD1GPU=A100, (B, T)=(8, 1024), Model size=GPT-2 small2026.0387,0001.85—
MuonGPU=A100, (B, T)=(12, 2048), Model size=GPT-2 small2026.0380,300——
AdamWGPU=MI250, (B, T)=(12, 2048), Model size=GPT-2 small2026.0373,700—1.15
MUD1GPU=MI250, (B, T)=(12, 2048), Model size=GPT-2 small2026.0368,0001.06—
AdamWGPU=MI250, (B, T)=(8, 4096), Model size=GPT-2 small2026.0364,900—1.13
MuonGPU=MI250, (B, T)=(12, 2048), Model size=GPT-2 small2026.0364,000——
MUD1GPU=MI250, (B, T)=(8, 4096), Model size=GPT-2 small2026.0362,4001.08—
MuonGPU=MI250, (B, T)=(8, 4096), Model size=GPT-2 small2026.0357,600——
MuonGPU=A100, (B, T)=(8, 1024), Model size=GPT-2 small2026.0346,900——
AdamWGPU=MI250, (B, T)=(8, 1024), Model size=GPT-2 small2026.0328,200—1.89
MUD1GPU=MI250, (B, T)=(8, 1024), Model size=GPT-2 small2026.0323,0001.54—
MuonGPU=MI250, (B, T)=(8, 1024), Model size=GPT-2 small2026.0314,900——