List your company
ResearchFormerly WizWand

Language Modeling on FineWeb-Edu (Throughput and Speedup)

71,600Throughput (tokens/s)AdamW
1,81619,93338,05056,167Mar 18, 2026
Updated 7mo ago

Evaluation results

MethodLinks
AdamWGPU=GH200, (B, T)=(8, 2048), Model=GPT-2 large2026.0371,600—3.43
AdamWGPU=GH200, (B, T)=(8, 1024), Model=GPT-2 large2026.0370,700—5.89
AdamWGPU=GH200, (B, T)=(8, 4096), Model=GPT-2 large2026.0366,400—2.11
MUD1GPU=GH200, (B, T)=(8, 4096), Model=GPT-2 large2026.0352,2001.66—
MUD1GPU=GH200, (B, T)=(8, 2048), Model=GPT-2 large2026.0344,5002.13—
MuonGPU=GH200, (B, T)=(8, 4096), Model=GPT-2 large2026.0331,400——
MUD1GPU=GH200, (B, T)=(8, 1024), Model=GPT-2 large2026.0330,6002.55—
AdamWGPU=A100, (B, T)=(8, 1024), Model=GPT-2 large2026.0325,000—5.56
MuonGPU=GH200, (B, T)=(8, 2048), Model=GPT-2 large2026.0320,900——
AdamWGPU=MI250, (B, T)=(8, 1024), Model=GPT-2 large2026.0313,900—2.24
MUD1GPU=A100, (B, T)=(8, 1024), Model=GPT-2 large2026.0313,2002.93—
MuonGPU=GH200, (B, T)=(8, 1024), Model=GPT-2 large2026.0312,000——
MUD1GPU=MI250, (B, T)=(8, 1024), Model=GPT-2 large2026.0310,9001.76—
MuonGPU=MI250, (B, T)=(8, 1024), Model=GPT-2 large2026.036,200——
MuonGPU=A100, (B, T)=(8, 1024), Model=GPT-2 large2026.034,500——