List your company
ResearchFormerly WizWand

Causal Language Model Pre-training on OpenWebText

15.24Throughput (samples/GPU/s)PACI
0.18084.0904811.9096Jun 5, 2026
Updated 4mo ago

Evaluation results

MethodLinks
PACIModel Scale=1.5B, #GPU=8, #Microbatch=242026.0615.2430
PACIModel Scale=1.5B, #GPU=8, #Microbatch=322026.0615.2330
PACIModel Scale=1.5B, #GPU=8, #Microbatch=642026.0615.0930
ZB-2pModel Scale=1.5B, #GPU=8, #Microbatch=642026.0614.959
ZB-2pModel Scale=1.5B, #GPU=8, #Microbatch=322026.0614.859
ZB-2pModel Scale=1.5B, #GPU=8, #Microbatch=242026.0614.559
ZB-1pModel Scale=1.5B, #GPU=8, #Microbatch=642026.0614.232
1F1B-IModel Scale=1.5B, #GPU=8, #Microbatch=642026.0613.940
1F1B-FlushModel Scale=1.5B, #GPU=8, #Microbatch=642026.0613.630
ZB-1pModel Scale=1.5B, #GPU=8, #Microbatch=322026.0613.432
1F1B-IModel Scale=1.5B, #GPU=8, #Microbatch=322026.0613.440
1F1B-IModel Scale=1.5B, #GPU=8, #Microbatch=242026.0613.140
ZB-1pModel Scale=1.5B, #GPU=8, #Microbatch=242026.0612.932
1F1B-FlushModel Scale=1.5B, #GPU=8, #Microbatch=322026.0612.530
1F1B-FlushModel Scale=1.5B, #GPU=8, #Microbatch=242026.0611.830
PACIModel Scale=6.2B, #GPU=8, #Microbatch=242026.064.5239
PACIModel Scale=6.2B, #GPU=8, #Microbatch=322026.064.5139
PACIModel Scale=6.2B, #GPU=8, #Microbatch=642026.064.4739
ZB-2pModel Scale=6.2B, #GPU=8, #Microbatch=642026.064.3970
ZB-2pModel Scale=6.2B, #GPU=8, #Microbatch=322026.064.3570
ZB-2pModel Scale=6.2B, #GPU=8, #Microbatch=242026.064.3270
ZB-1pModel Scale=6.2B, #GPU=8, #Microbatch=642026.064.242
1F1B-IModel Scale=6.2B, #GPU=8, #Microbatch=642026.064.1948
1F1B-IModel Scale=6.2B, #GPU=8, #Microbatch=322026.064.0848
1F1B-FlushModel Scale=6.2B, #GPU=8, #Microbatch=642026.064.0339
1F1B-IModel Scale=6.2B, #GPU=8, #Microbatch=242026.064.0148
ZB-1pModel Scale=6.2B, #GPU=8, #Microbatch=322026.06442
ZB-1pModel Scale=6.2B, #GPU=8, #Microbatch=242026.063.8842
1F1B-FlushModel Scale=6.2B, #GPU=8, #Microbatch=322026.063.739
1F1B-FlushModel Scale=6.2B, #GPU=8, #Microbatch=242026.063.539
ZB-2pModel Scale=14.6B, #GPU=16, #Microbatch=1282026.061.8551
PACIModel Scale=14.6B, #GPU=16, #Microbatch=482026.061.8432
PACIModel Scale=14.6B, #GPU=16, #Microbatch=642026.061.8432
ZB-2pModel Scale=14.6B, #GPU=16, #Microbatch=642026.061.8351
PACIModel Scale=14.6B, #GPU=16, #Microbatch=1282026.061.8332
ZB-2pModel Scale=14.6B, #GPU=16, #Microbatch=482026.061.8151
ZB-1pModel Scale=14.6B, #GPU=16, #Microbatch=1282026.061.7633
ZB-1pModel Scale=14.6B, #GPU=16, #Microbatch=642026.061.6733
1F1B-IModel Scale=14.6B, #GPU=16, #Microbatch=1282026.061.6639
1F1B-FlushModel Scale=14.6B, #GPU=16, #Microbatch=1282026.061.6432
ZB-1pModel Scale=14.6B, #GPU=16, #Microbatch=482026.061.6133
1F1B-IModel Scale=14.6B, #GPU=16, #Microbatch=642026.061.5939
1F1B-IModel Scale=14.6B, #GPU=16, #Microbatch=482026.061.5439
1F1B-FlushModel Scale=14.6B, #GPU=16, #Microbatch=642026.061.4932
1F1B-FlushModel Scale=14.6B, #GPU=16, #Microbatch=482026.061.432
PACIModel Scale=28.3B, #GPU=32, #Microbatch=962026.061.0143
ZB-2pModel Scale=28.3B, #GPU=32, #Microbatch=1282026.06174
ZB-2pModel Scale=28.3B, #GPU=32, #Microbatch=2562026.06174
ZB-2pModel Scale=28.3B, #GPU=32, #Microbatch=962026.060.9974
PACIModel Scale=28.3B, #GPU=32, #Microbatch=1282026.060.9943
PACIModel Scale=28.3B, #GPU=32, #Microbatch=2562026.060.9943
ZB-1pModel Scale=28.3B, #GPU=32, #Microbatch=2562026.060.9644
ZB-1pModel Scale=28.3B, #GPU=32, #Microbatch=1282026.060.944
1F1B-IModel Scale=28.3B, #GPU=32, #Microbatch=2562026.060.958
1F1B-FlushModel Scale=28.3B, #GPU=32, #Microbatch=2562026.060.8843
ZB-1pModel Scale=28.3B, #GPU=32, #Microbatch=962026.060.8744
1F1B-IModel Scale=28.3B, #GPU=32, #Microbatch=1282026.060.8558
1F1B-IModel Scale=28.3B, #GPU=32, #Microbatch=962026.060.8258
1F1B-FlushModel Scale=28.3B, #GPU=32, #Microbatch=1282026.060.843
1F1B-FlushModel Scale=28.3B, #GPU=32, #Microbatch=962026.060.7643