LLM Training Performance on Alpaca Llama 3.3 70B
74.75Memory Usage (GB/GPU)torchtune + AC
Evaluation results
| Method | Links | ||
|---|---|---|---|
| torchtune + ACModel=Llama 3.3 70B, Optimizations=Activation Checkpointing (AC), Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 74.75 | 122.55 | |
| torchtune + Optim BwdModel=Llama 3.3 70B, Optimizations=Optimizer in backward, Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=162026.05 | 74.89 | 352.11 | |
| torchtune + CompileModel=Llama 3.3 70B, Optimizations=torch.compile (excluding optimizer step), Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 75.22 | 128.57 |