Distributed Training on LLaMA2 13B
0Number of Recomputed LayersAGoQ
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AGoQSequence Length=32K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 36,568 | |
| AGoQSequence Length=40K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 45,590 | |
| AGoQSequence Length=48K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 57,047 | |
| AGoQSequence Length=56K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 69,544 | |
| AGoQSequence Length=64K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 82,519 | |
| AGoQSequence Length=72K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 97,615 | |
| AGoQSequence Length=80K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 0 | 111,422 | |
| ZeRO-1Sequence Length=32K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 2 | 37,038 | |
| Megatron-LMSequence Length=32K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 3 | 37,635 | |
| Megatron-LMSequence Length=40K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 4 | 51,922 | |
| ZeRO-1Sequence Length=40K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 4 | 51,418 | |
| Megatron-LMSequence Length=48K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 6 | 67,932 | |
| ZeRO-1Sequence Length=48K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 6 | 67,928 | |
| Megatron-LMSequence Length=56K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 8 | 88,200 | |
| ZeRO-1Sequence Length=56K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 8 | 86,601 | |
| Megatron-LMSequence Length=64K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 8 | 104,444 | |
| ZeRO-1Sequence Length=64K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 8 | 103,706 | |
| Megatron-LMSequence Length=72K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 10 | 128,085 | |
| ZeRO-1Sequence Length=72K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 10 | 128,152 | |
| Megatron-LMSequence Length=80K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 10 | 149,667 | |
| ZeRO-1Sequence Length=80K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05 | 10 | 149,288 |