LLM Training on Llama2-70B (64 x H100-8)
7.8Iteration Time (s)Megatron-LM
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Megatron-LMHardware=64 x H100-82025.07 | 7.8 | 47.2 | — | |
| AXLearnHardware=64 x H100-82025.07 | 9.2 | 40 | — | |
| MaxTextHardware=64 x H100-82025.07 | 9.4 | 39.1 | — | |
| PyTorch FSDPHardware=64 x H100-82025.07 | 10.6 | 34.7 | — |