List your company
ResearchFormerly WizWand

Weak Scaling on BERT-Base (train)

3,707.01Memory (MB)Sequence parallelism
3,260.80686,272.67849,284.5512,296.4216May 26, 2021
Updated 8mo ago

Evaluation results

MethodLinks
Sequence parallelismParallel size=1, Batch size=64, Sequence length=2562021.053,707.019,340.13
Tensor parallelismParallel size=1, Batch size=64, Sequence length=2562021.053,707.399,752.61
Sequence parallelismParallel size=2, Batch size=64, Sequence length=5122021.054,670.6413,144.16
Tensor parallelismParallel size=2, Batch size=64, Sequence length=5122021.054,993.4314,195.17
Sequence parallelismParallel size=4, Batch size=64, Sequence length=10242021.056,601.8818,243.82
Tensor parallelismParallel size=4, Batch size=64, Sequence length=10242021.058,175.9319,879.27
Tensor parallelismParallel size=1, Batch size=64, Sequence length=5122021.058,477.289,946.15
Sequence parallelismParallel size=1, Batch size=64, Sequence length=5122021.058,477.539,261.04
Sequence parallelismParallel size=2, Batch size=128, Sequence length=5122021.058,478.7613,938.22
Sequence parallelismParallel size=4, Batch size=256, Sequence length=5122021.058,481.2621,269.91
Sequence parallelismParallel size=8, Batch size=512, Sequence length=5122021.058,490.7526,401.64
Tensor parallelismParallel size=2, Batch size=128, Sequence length=5122021.059,520.4715,510.19
Sequence parallelismParallel size=8, Batch size=64, Sequence length=20482021.0510,536.3821,625.51
Tensor parallelismParallel size=4, Batch size=256, Sequence length=5122021.0512,232.5220,701.96
Tensor parallelismParallel size=8, Batch size=64, Sequence length=20482021.0514,862.0922,330.5