Weak Scaling on BERT-Base (train)
3,707.01Memory (MB)Sequence parallelism
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Sequence parallelismParallel size=1, Batch size=64, Sequence length=2562021.05 | 3,707.01 | 9,340.13 | |
| Tensor parallelismParallel size=1, Batch size=64, Sequence length=2562021.05 | 3,707.39 | 9,752.61 | |
| Sequence parallelismParallel size=2, Batch size=64, Sequence length=5122021.05 | 4,670.64 | 13,144.16 | |
| Tensor parallelismParallel size=2, Batch size=64, Sequence length=5122021.05 | 4,993.43 | 14,195.17 | |
| Sequence parallelismParallel size=4, Batch size=64, Sequence length=10242021.05 | 6,601.88 | 18,243.82 | |
| Tensor parallelismParallel size=4, Batch size=64, Sequence length=10242021.05 | 8,175.93 | 19,879.27 | |
| Tensor parallelismParallel size=1, Batch size=64, Sequence length=5122021.05 | 8,477.28 | 9,946.15 | |
| Sequence parallelismParallel size=1, Batch size=64, Sequence length=5122021.05 | 8,477.53 | 9,261.04 | |
| Sequence parallelismParallel size=2, Batch size=128, Sequence length=5122021.05 | 8,478.76 | 13,938.22 | |
| Sequence parallelismParallel size=4, Batch size=256, Sequence length=5122021.05 | 8,481.26 | 21,269.91 | |
| Sequence parallelismParallel size=8, Batch size=512, Sequence length=5122021.05 | 8,490.75 | 26,401.64 | |
| Tensor parallelismParallel size=2, Batch size=128, Sequence length=5122021.05 | 9,520.47 | 15,510.19 | |
| Sequence parallelismParallel size=8, Batch size=64, Sequence length=20482021.05 | 10,536.38 | 21,625.51 | |
| Tensor parallelismParallel size=4, Batch size=256, Sequence length=5122021.05 | 12,232.52 | 20,701.96 | |
| Tensor parallelismParallel size=8, Batch size=64, Sequence length=20482021.05 | 14,862.09 | 22,330.5 |