Language Model Training Efficiency on FineWeb
3,642.88ThroughputHSAP
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| HSAPContext Length=32K, Topology (intraSPxinterSP)=1x82026.06 | 3,642.88 | 0.99 | |
| HSAPContext Length=32K, Topology (intraSPxinterSP)=8x12026.06 | 3,598.57 | 0.98 | |
| HSAPContext Length=32K, Topology (intraSPxinterSP)=4x22026.06 | 3,574.18 | 0.97 | |
| HSAPContext Length=32K, Topology (intraSPxinterSP)=2x42026.06 | 3,510.77 | 0.95 | |
| HSAPContext Length=64K, Topology (intraSPxinterSP)=8x12026.06 | 3,459.34 | 0.47 | |
| HSAPContext Length=64K, Topology (intraSPxinterSP)=2x42026.06 | 3,267.28 | 0.44 | |
| HSAPContext Length=64K, Topology (intraSPxinterSP)=4x22026.06 | 3,250.4 | 0.44 | |
| HSAPContext Length=64K, Topology (intraSPxinterSP)=1x82026.06 | 3,100.68 | 0.42 | |
| PackingContext Length=16K, Topology (intraSPxinterSP)=8x12026.06 | 2,883.24 | 1.56 | |
| BatchingContext Length=16K, Topology (intraSPxinterSP)=8x12026.06 | 2,877.61 | 1.56 | |
| HSAPContext Length=16K, Topology (intraSPxinterSP)=8x12026.06 | 2,863.02 | 1.55 | |
| PackingContext Length=32K, Topology (intraSPxinterSP)=8x12026.06 | 2,519.64 | 0.68 | |
| PackingContext Length=16K, Topology (intraSPxinterSP)=4x22026.06 | 2,485.24 | 1.35 | |
| HSAPContext Length=16K, Topology (intraSPxinterSP)=4x22026.06 | 2,480.91 | 1.35 | |
| BatchingContext Length=16K, Topology (intraSPxinterSP)=4x22026.06 | 2,475.36 | 1.34 | |
| BatchingContext Length=16K, Topology (intraSPxinterSP)=2x42026.06 | 2,442.83 | 1.33 | |
| HSAPContext Length=16K, Topology (intraSPxinterSP)=2x42026.06 | 2,435.17 | 1.32 | |
| PackingContext Length=16K, Topology (intraSPxinterSP)=2x42026.06 | 2,430.87 | 1.32 | |
| PackingContext Length=32K, Topology (intraSPxinterSP)=2x42026.06 | 2,121.79 | 0.58 | |
| PackingContext Length=32K, Topology (intraSPxinterSP)=1x82026.06 | 2,067.07 | 0.56 | |
| PackingContext Length=32K, Topology (intraSPxinterSP)=4x22026.06 | 2,060.45 | 0.56 | |
| HSAPContext Length=16K, Topology (intraSPxinterSP)=1x82026.06 | 2,002.1 | 1.09 | |
| PackingContext Length=16K, Topology (intraSPxinterSP)=1x82026.06 | 1,998.31 | 1.08 | |
| BatchingContext Length=16K, Topology (intraSPxinterSP)=1x82026.06 | 1,997.5 | 1.08 | |
| PackingContext Length=64K, Topology (intraSPxinterSP)=8x12026.06 | 1,900.28 | 0.26 | |
| PackingContext Length=64K, Topology (intraSPxinterSP)=1x82026.06 | 1,622 | 0.22 | |
| PackingContext Length=64K, Topology (intraSPxinterSP)=2x42026.06 | 1,611.48 | 0.22 | |
| PackingContext Length=64K, Topology (intraSPxinterSP)=4x22026.06 | 1,482.95 | 0.2 | |
| BatchingContext Length=32K, Topology (intraSPxinterSP)=8x12026.06 | 1,277.91 | 0.69 | |
| BatchingContext Length=32K, Topology (intraSPxinterSP)=2x42026.06 | 1,070.83 | 0.58 | |
| BatchingContext Length=32K, Topology (intraSPxinterSP)=4x22026.06 | 1,040.75 | 0.56 | |
| BatchingContext Length=32K, Topology (intraSPxinterSP)=1x82026.06 | 1,037.41 | 0.56 | |
| BatchingContext Length=64K, Topology (intraSPxinterSP)=8x12026.06 | 114.41 | 0.26 | |
| BatchingContext Length=64K, Topology (intraSPxinterSP)=1x82026.06 | 97.67 | 0.22 | |
| BatchingContext Length=64K, Topology (intraSPxinterSP)=2x42026.06 | 96.83 | 0.22 | |
| BatchingContext Length=64K, Topology (intraSPxinterSP)=4x22026.06 | 88.79 | 0.2 |