Training Memory Usage Profiling on Llama3-8B 8×H100s
21.1Peak Memory Usage (128K)UPipe
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| UPipeContext Parallelism Scheme=UPipe, Unit=GiB2026.02 | 21.1 | 22.3 | 24.7 | 29.9 | 40.5 | 51.1 | 61.7 | 72.3 | |
| UlyssesContext Parallelism Scheme=Ulysses, Unit=GiB2026.02 | 21.26 | 23.02 | 26.8 | 34.35 | 49.49 | 64.55 | — | — | |
| RingContext Parallelism Scheme=Ring, Unit=GiB2026.02 | 21.32 | 23.4 | 27.58 | 35.86 | 52.49 | 69.11 | — | — | |
| FPDTContext Parallelism Scheme=FPDT, Unit=GiB2026.02 | 21.73 | 22.5 | 24.03 | 27.09 | 35.17 | 43.35 | 51.42 | — | |
| Native-PyTorchContext Parallelism Scheme=Native-PyTorch, Unit=GiB2026.02 | 25.32 | 31.4 | 43.55 | 67.86 | — | — | — | — |