Language Modeling on NanoGPT OpenWebText
391,100Throughput (tokens/s)AdamW
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| AdamWGPU=GH200, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 391,100 | — | 1.82 | |
| AdamWGPU=GH200, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 342,400 | — | 3.14 | |
| MUD1GPU=GH200, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 295,300 | 1.38 | — | |
| MuonGPU=GH200, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 214,600 | — | — | |
| MUD1GPU=GH200, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 185,800 | 1.7 | — | |
| AdamWGPU=A100, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 131,800 | — | 2.81 | |
| AdamWGPU=A100, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 130,000 | — | 1.62 | |
| AdamWGPU=A100, (B, T)=(8, 4096), Model size=GPT-2 small2026.03 | 125,700 | — | 1.43 | |
| MUD1GPU=A100, (B, T)=(8, 4096), Model size=GPT-2 small2026.03 | 113,700 | 1.29 | — | |
| MuonGPU=GH200, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 109,000 | — | — | |
| MUD1GPU=A100, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 108,000 | 1.35 | — | |
| MuonGPU=A100, (B, T)=(8, 4096), Model size=GPT-2 small2026.03 | 88,000 | — | — | |
| MUD1GPU=A100, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 87,000 | 1.85 | — | |
| MuonGPU=A100, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 80,300 | — | — | |
| AdamWGPU=MI250, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 73,700 | — | 1.15 | |
| MUD1GPU=MI250, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 68,000 | 1.06 | — | |
| AdamWGPU=MI250, (B, T)=(8, 4096), Model size=GPT-2 small2026.03 | 64,900 | — | 1.13 | |
| MuonGPU=MI250, (B, T)=(12, 2048), Model size=GPT-2 small2026.03 | 64,000 | — | — | |
| MUD1GPU=MI250, (B, T)=(8, 4096), Model size=GPT-2 small2026.03 | 62,400 | 1.08 | — | |
| MuonGPU=MI250, (B, T)=(8, 4096), Model size=GPT-2 small2026.03 | 57,600 | — | — | |
| MuonGPU=A100, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 46,900 | — | — | |
| AdamWGPU=MI250, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 28,200 | — | 1.89 | |
| MUD1GPU=MI250, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 23,000 | 1.54 | — | |
| MuonGPU=MI250, (B, T)=(8, 1024), Model size=GPT-2 small2026.03 | 14,900 | — | — |