List your company
ResearchFormerly WizWand

Distributed Training on LLaMA2 13B

0Number of Recomputed LayersAGoQ
-0.42.357.7May 1, 2026
Updated 6mo ago

Evaluation results

MethodLinks
AGoQSequence Length=32K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05036,568
AGoQSequence Length=40K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05045,590
AGoQSequence Length=48K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05057,047
AGoQSequence Length=56K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05069,544
AGoQSequence Length=64K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05082,519
AGoQSequence Length=72K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05097,615
AGoQSequence Length=80K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.050111,422
ZeRO-1Sequence Length=32K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05237,038
Megatron-LMSequence Length=32K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05337,635
Megatron-LMSequence Length=40K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05451,922
ZeRO-1Sequence Length=40K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05451,418
Megatron-LMSequence Length=48K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05667,932
ZeRO-1Sequence Length=48K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05667,928
Megatron-LMSequence Length=56K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05888,200
ZeRO-1Sequence Length=56K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.05886,601
Megatron-LMSequence Length=64K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.058104,444
ZeRO-1Sequence Length=64K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.058103,706
Megatron-LMSequence Length=72K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.0510128,085
ZeRO-1Sequence Length=72K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.0510128,152
Megatron-LMSequence Length=80K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.0510149,667
ZeRO-1Sequence Length=80K, Mini-batch size=1, Global batch size=16, Pipeline Parallelism (PP)=4, Tensor Parallelism (TP)=82026.0510149,288