LLM Training Performance on Alpaca Qwen3 32B
38.43Memory (GB/GPU)torchtune + AC + LCE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| torchtune + AC + LCEModel=Qwen3 32B, Optimizations=AC, Linear Cross Entropy (LCE), Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 38.43 | 398.56 | |
| AxolotlModel=Qwen3 32B, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 40.9 | 218 | |
| torchtune + ACModel=Qwen3 32B, Optimizations=Activation Checkpointing (AC), Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 42.93 | 405.95 | |
| torchtune + AC + LCE + CompileModel=Qwen3 32B, Optimizations=AC, LCE, torch.compile, Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 44.17 | 433.12 | |
| torchtune + AC + LCE + Compile + Optim BwdModel=Qwen3 32B, Optimizations=AC, LCE, torch.compile, Optimizer in backward, Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=162026.05 | 60.41 | 581.63 | |
| torchtuneModel=Qwen3 32B, Optimizations=None, Parallelism=FSDP2 with tensor, data, and loss parallelism, Hardware=1x8 H100s, Sequence Length=2048, Micro-batch size=22026.05 | 67.78 | 465.79 |