Training Speed Measurement on LLM Sequences (train)
13.4Step Time (ms)HSAP
Evaluation Results
| Method | Links | |
|---|---|---|
| HSAPSequence Per GPU=4K, GPUs=82026.06 | 13.4 | |
| Deepspeed-UlyssesSequence Per GPU=4K, GPUs=82026.06 | 14.6 | |
| DistFlashAttnSequence Per GPU=4K, GPUs=82026.06 | 20.46 | |
| HSAPSequence Per GPU=4K, GPUs=162026.06 | 26.87 | |
| Deepspeed-UlyssesSequence Per GPU=4K, GPUs=162026.06 | 29.31 | |
| HSAPSequence Per GPU=8K, GPUs=82026.06 | 45.88 | |
| Deepspeed-UlyssesSequence Per GPU=8K, GPUs=82026.06 | 49.74 | |
| DistFlashAttnSequence Per GPU=4K, GPUs=162026.06 | 54.58 | |
| DistFlashAttnSequence Per GPU=8K, GPUs=82026.06 | 60.57 | |
| HSAPSequence Per GPU=8K, GPUs=162026.06 | 92.54 | |
| Deepspeed-UlyssesSequence Per GPU=8K, GPUs=162026.06 | 100.42 | |
| HSAPSequence Per GPU=4K, GPUs=642026.06 | 116.68 | |
| DistFlashAttnSequence Per GPU=8K, GPUs=162026.06 | 136.99 | |
| HSAPSequence Per GPU=16K, GPUs=82026.06 | 168.44 | |
| Deepspeed-UlyssesSequence Per GPU=16K, GPUs=82026.06 | 176.23 | |
| DistFlashAttnSequence Per GPU=16K, GPUs=82026.06 | 205.29 | |
| DistFlashAttnSequence Per GPU=4K, GPUs=642026.06 | 282.82 | |
| HSAPSequence Per GPU=16K, GPUs=162026.06 | 335.71 | |
| Deepspeed-UlyssesSequence Per GPU=16K, GPUs=162026.06 | 349.85 | |
| HSAPSequence Per GPU=8K, GPUs=642026.06 | 436.45 | |
| DistFlashAttnSequence Per GPU=16K, GPUs=162026.06 | 436.65 | |
| DistFlashAttnSequence Per GPU=8K, GPUs=642026.06 | 613.12 | |
| HSAPSequence Per GPU=32K, GPUs=82026.06 | 662.98 | |
| Deepspeed-UlyssesSequence Per GPU=32K, GPUs=82026.06 | 664.33 | |
| DistFlashAttnSequence Per GPU=32K, GPUs=82026.06 | 763.13 | |
| HSAPSequence Per GPU=32K, GPUs=162026.06 | 1,350.28 | |
| Deepspeed-UlyssesSequence Per GPU=32K, GPUs=162026.06 | 1,367.85 | |
| DistFlashAttnSequence Per GPU=32K, GPUs=162026.06 | 1,545.35 | |
| HSAPSequence Per GPU=16K, GPUs=642026.06 | 1,645.43 | |
| DistFlashAttnSequence Per GPU=16K, GPUs=642026.06 | 1,788.33 | |
| HSAPSequence Per GPU=32K, GPUs=642026.06 | 6,156.32 | |
| DistFlashAttnSequence Per GPU=32K, GPUs=642026.06 | 6,427.15 |