Language Modeling on FineWeb-Edu (Throughput and Speedup)
71,600Throughput (tokens/s)AdamW
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| AdamWGPU=GH200, (B, T)=(8, 2048), Model=GPT-2 large2026.03 | 71,600 | — | 3.43 | |
| AdamWGPU=GH200, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 70,700 | — | 5.89 | |
| AdamWGPU=GH200, (B, T)=(8, 4096), Model=GPT-2 large2026.03 | 66,400 | — | 2.11 | |
| MUD1GPU=GH200, (B, T)=(8, 4096), Model=GPT-2 large2026.03 | 52,200 | 1.66 | — | |
| MUD1GPU=GH200, (B, T)=(8, 2048), Model=GPT-2 large2026.03 | 44,500 | 2.13 | — | |
| MuonGPU=GH200, (B, T)=(8, 4096), Model=GPT-2 large2026.03 | 31,400 | — | — | |
| MUD1GPU=GH200, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 30,600 | 2.55 | — | |
| AdamWGPU=A100, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 25,000 | — | 5.56 | |
| MuonGPU=GH200, (B, T)=(8, 2048), Model=GPT-2 large2026.03 | 20,900 | — | — | |
| AdamWGPU=MI250, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 13,900 | — | 2.24 | |
| MUD1GPU=A100, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 13,200 | 2.93 | — | |
| MuonGPU=GH200, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 12,000 | — | — | |
| MUD1GPU=MI250, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 10,900 | 1.76 | — | |
| MuonGPU=MI250, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 6,200 | — | — | |
| MuonGPU=A100, (B, T)=(8, 1024), Model=GPT-2 large2026.03 | 4,500 | — | — |