List your company
ResearchFormerly WizWand

Language Model Training Efficiency on FineWeb

3,642.88ThroughputHSAP
-53.3736906.23071,865.8352,825.4393Jun 29, 2026
Updated 4mo ago

Evaluation results

MethodLinks
HSAPContext Length=32K, Topology (intraSPxinterSP)=1x82026.063,642.880.99
HSAPContext Length=32K, Topology (intraSPxinterSP)=8x12026.063,598.570.98
HSAPContext Length=32K, Topology (intraSPxinterSP)=4x22026.063,574.180.97
HSAPContext Length=32K, Topology (intraSPxinterSP)=2x42026.063,510.770.95
HSAPContext Length=64K, Topology (intraSPxinterSP)=8x12026.063,459.340.47
HSAPContext Length=64K, Topology (intraSPxinterSP)=2x42026.063,267.280.44
HSAPContext Length=64K, Topology (intraSPxinterSP)=4x22026.063,250.40.44
HSAPContext Length=64K, Topology (intraSPxinterSP)=1x82026.063,100.680.42
PackingContext Length=16K, Topology (intraSPxinterSP)=8x12026.062,883.241.56
BatchingContext Length=16K, Topology (intraSPxinterSP)=8x12026.062,877.611.56
HSAPContext Length=16K, Topology (intraSPxinterSP)=8x12026.062,863.021.55
PackingContext Length=32K, Topology (intraSPxinterSP)=8x12026.062,519.640.68
PackingContext Length=16K, Topology (intraSPxinterSP)=4x22026.062,485.241.35
HSAPContext Length=16K, Topology (intraSPxinterSP)=4x22026.062,480.911.35
BatchingContext Length=16K, Topology (intraSPxinterSP)=4x22026.062,475.361.34
BatchingContext Length=16K, Topology (intraSPxinterSP)=2x42026.062,442.831.33
HSAPContext Length=16K, Topology (intraSPxinterSP)=2x42026.062,435.171.32
PackingContext Length=16K, Topology (intraSPxinterSP)=2x42026.062,430.871.32
PackingContext Length=32K, Topology (intraSPxinterSP)=2x42026.062,121.790.58
PackingContext Length=32K, Topology (intraSPxinterSP)=1x82026.062,067.070.56
PackingContext Length=32K, Topology (intraSPxinterSP)=4x22026.062,060.450.56
HSAPContext Length=16K, Topology (intraSPxinterSP)=1x82026.062,002.11.09
PackingContext Length=16K, Topology (intraSPxinterSP)=1x82026.061,998.311.08
BatchingContext Length=16K, Topology (intraSPxinterSP)=1x82026.061,997.51.08
PackingContext Length=64K, Topology (intraSPxinterSP)=8x12026.061,900.280.26
PackingContext Length=64K, Topology (intraSPxinterSP)=1x82026.061,6220.22
PackingContext Length=64K, Topology (intraSPxinterSP)=2x42026.061,611.480.22
PackingContext Length=64K, Topology (intraSPxinterSP)=4x22026.061,482.950.2
BatchingContext Length=32K, Topology (intraSPxinterSP)=8x12026.061,277.910.69
BatchingContext Length=32K, Topology (intraSPxinterSP)=2x42026.061,070.830.58
BatchingContext Length=32K, Topology (intraSPxinterSP)=4x22026.061,040.750.56
BatchingContext Length=32K, Topology (intraSPxinterSP)=1x82026.061,037.410.56
BatchingContext Length=64K, Topology (intraSPxinterSP)=8x12026.06114.410.26
BatchingContext Length=64K, Topology (intraSPxinterSP)=1x82026.0697.670.22
BatchingContext Length=64K, Topology (intraSPxinterSP)=2x42026.0696.830.22
BatchingContext Length=64K, Topology (intraSPxinterSP)=4x22026.0688.790.2