Training Memory Usage Profiling on Qwen3-32B 16×H100s
38.94Memory Footprint (Seq 128K)FPDT
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| FPDTContext Parallelism Scheme=FPDT, Unit=GiB2026.02 | 38.94 | 39.47 | 40.54 | 42.66 | 46.91 | 52.27 | 57.77 | — | |
| UPipeContext Parallelism Scheme=UPipe, Unit=GiB2026.02 | 39.98 | 40.84 | 42.72 | 46.84 | 55.65 | 64.47 | 73.28 | — | |
| UlyssesContext Parallelism Scheme=Ulysses, Unit=GiB2026.02 | 40.13 | 41.16 | 44.1 | 50.27 | 62.6 | — | — | — | |
| RingContext Parallelism Scheme=Ring, Unit=GiB2026.02 | 40.14 | 41.16 | 44.22 | 50.51 | 63.11 | — | — | — | |
| Native-PyTorchContext Parallelism Scheme=Native-PyTorch, Unit=GiB2026.02 | 45.81 | 53.69 | 69.47 | — | — | — | — | — |