LLM Training on Llama2 7B
1.4Iteration Time (s)MaxText
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MaxTextHardware=32 x H100-82025.07 | 1.4 | 54.7 | — | |
| AXLearnHardware=32 x H100-82025.07 | 1.4 | 54.2 | — | |
| Megatron-LMHardware=32 x H100-82025.07 | 1.7 | 44.9 | — | |
| PyTorch FSDPHardware=32 x H100-82025.07 | 2.6 | 29.9 | — |