Causal Language Model Pre-training on OpenWebText
15.24Throughput (samples/GPU/s)PACI
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| PACIModel Scale=1.5B, #GPU=8, #Microbatch=242026.06 | 15.24 | 30 | |
| PACIModel Scale=1.5B, #GPU=8, #Microbatch=322026.06 | 15.23 | 30 | |
| PACIModel Scale=1.5B, #GPU=8, #Microbatch=642026.06 | 15.09 | 30 | |
| ZB-2pModel Scale=1.5B, #GPU=8, #Microbatch=642026.06 | 14.9 | 59 | |
| ZB-2pModel Scale=1.5B, #GPU=8, #Microbatch=322026.06 | 14.8 | 59 | |
| ZB-2pModel Scale=1.5B, #GPU=8, #Microbatch=242026.06 | 14.5 | 59 | |
| ZB-1pModel Scale=1.5B, #GPU=8, #Microbatch=642026.06 | 14.2 | 32 | |
| 1F1B-IModel Scale=1.5B, #GPU=8, #Microbatch=642026.06 | 13.9 | 40 | |
| 1F1B-FlushModel Scale=1.5B, #GPU=8, #Microbatch=642026.06 | 13.6 | 30 | |
| ZB-1pModel Scale=1.5B, #GPU=8, #Microbatch=322026.06 | 13.4 | 32 | |
| 1F1B-IModel Scale=1.5B, #GPU=8, #Microbatch=322026.06 | 13.4 | 40 | |
| 1F1B-IModel Scale=1.5B, #GPU=8, #Microbatch=242026.06 | 13.1 | 40 | |
| ZB-1pModel Scale=1.5B, #GPU=8, #Microbatch=242026.06 | 12.9 | 32 | |
| 1F1B-FlushModel Scale=1.5B, #GPU=8, #Microbatch=322026.06 | 12.5 | 30 | |
| 1F1B-FlushModel Scale=1.5B, #GPU=8, #Microbatch=242026.06 | 11.8 | 30 | |
| PACIModel Scale=6.2B, #GPU=8, #Microbatch=242026.06 | 4.52 | 39 | |
| PACIModel Scale=6.2B, #GPU=8, #Microbatch=322026.06 | 4.51 | 39 | |
| PACIModel Scale=6.2B, #GPU=8, #Microbatch=642026.06 | 4.47 | 39 | |
| ZB-2pModel Scale=6.2B, #GPU=8, #Microbatch=642026.06 | 4.39 | 70 | |
| ZB-2pModel Scale=6.2B, #GPU=8, #Microbatch=322026.06 | 4.35 | 70 | |
| ZB-2pModel Scale=6.2B, #GPU=8, #Microbatch=242026.06 | 4.32 | 70 | |
| ZB-1pModel Scale=6.2B, #GPU=8, #Microbatch=642026.06 | 4.2 | 42 | |
| 1F1B-IModel Scale=6.2B, #GPU=8, #Microbatch=642026.06 | 4.19 | 48 | |
| 1F1B-IModel Scale=6.2B, #GPU=8, #Microbatch=322026.06 | 4.08 | 48 | |
| 1F1B-FlushModel Scale=6.2B, #GPU=8, #Microbatch=642026.06 | 4.03 | 39 | |
| 1F1B-IModel Scale=6.2B, #GPU=8, #Microbatch=242026.06 | 4.01 | 48 | |
| ZB-1pModel Scale=6.2B, #GPU=8, #Microbatch=322026.06 | 4 | 42 | |
| ZB-1pModel Scale=6.2B, #GPU=8, #Microbatch=242026.06 | 3.88 | 42 | |
| 1F1B-FlushModel Scale=6.2B, #GPU=8, #Microbatch=322026.06 | 3.7 | 39 | |
| 1F1B-FlushModel Scale=6.2B, #GPU=8, #Microbatch=242026.06 | 3.5 | 39 | |
| ZB-2pModel Scale=14.6B, #GPU=16, #Microbatch=1282026.06 | 1.85 | 51 | |
| PACIModel Scale=14.6B, #GPU=16, #Microbatch=482026.06 | 1.84 | 32 | |
| PACIModel Scale=14.6B, #GPU=16, #Microbatch=642026.06 | 1.84 | 32 | |
| ZB-2pModel Scale=14.6B, #GPU=16, #Microbatch=642026.06 | 1.83 | 51 | |
| PACIModel Scale=14.6B, #GPU=16, #Microbatch=1282026.06 | 1.83 | 32 | |
| ZB-2pModel Scale=14.6B, #GPU=16, #Microbatch=482026.06 | 1.81 | 51 | |
| ZB-1pModel Scale=14.6B, #GPU=16, #Microbatch=1282026.06 | 1.76 | 33 | |
| ZB-1pModel Scale=14.6B, #GPU=16, #Microbatch=642026.06 | 1.67 | 33 | |
| 1F1B-IModel Scale=14.6B, #GPU=16, #Microbatch=1282026.06 | 1.66 | 39 | |
| 1F1B-FlushModel Scale=14.6B, #GPU=16, #Microbatch=1282026.06 | 1.64 | 32 | |
| ZB-1pModel Scale=14.6B, #GPU=16, #Microbatch=482026.06 | 1.61 | 33 | |
| 1F1B-IModel Scale=14.6B, #GPU=16, #Microbatch=642026.06 | 1.59 | 39 | |
| 1F1B-IModel Scale=14.6B, #GPU=16, #Microbatch=482026.06 | 1.54 | 39 | |
| 1F1B-FlushModel Scale=14.6B, #GPU=16, #Microbatch=642026.06 | 1.49 | 32 | |
| 1F1B-FlushModel Scale=14.6B, #GPU=16, #Microbatch=482026.06 | 1.4 | 32 | |
| PACIModel Scale=28.3B, #GPU=32, #Microbatch=962026.06 | 1.01 | 43 | |
| ZB-2pModel Scale=28.3B, #GPU=32, #Microbatch=1282026.06 | 1 | 74 | |
| ZB-2pModel Scale=28.3B, #GPU=32, #Microbatch=2562026.06 | 1 | 74 | |
| ZB-2pModel Scale=28.3B, #GPU=32, #Microbatch=962026.06 | 0.99 | 74 | |
| PACIModel Scale=28.3B, #GPU=32, #Microbatch=1282026.06 | 0.99 | 43 | |
| PACIModel Scale=28.3B, #GPU=32, #Microbatch=2562026.06 | 0.99 | 43 | |
| ZB-1pModel Scale=28.3B, #GPU=32, #Microbatch=2562026.06 | 0.96 | 44 | |
| ZB-1pModel Scale=28.3B, #GPU=32, #Microbatch=1282026.06 | 0.9 | 44 | |
| 1F1B-IModel Scale=28.3B, #GPU=32, #Microbatch=2562026.06 | 0.9 | 58 | |
| 1F1B-FlushModel Scale=28.3B, #GPU=32, #Microbatch=2562026.06 | 0.88 | 43 | |
| ZB-1pModel Scale=28.3B, #GPU=32, #Microbatch=962026.06 | 0.87 | 44 | |
| 1F1B-IModel Scale=28.3B, #GPU=32, #Microbatch=1282026.06 | 0.85 | 58 | |
| 1F1B-IModel Scale=28.3B, #GPU=32, #Microbatch=962026.06 | 0.82 | 58 | |
| 1F1B-FlushModel Scale=28.3B, #GPU=32, #Microbatch=1282026.06 | 0.8 | 43 | |
| 1F1B-FlushModel Scale=28.3B, #GPU=32, #Microbatch=962026.06 | 0.76 | 43 |