Scaling Efficiency on Transformer 128 tokens
93.29Scaling Efficiency (Linear Projection)AEGIS Parallel
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AEGIS ParallelInterconnect=NVLink, Number of GPUs=2, Input Tokens=1282026.04 | 93.29 | 92.37 | 89.32 | 94.21 | 94.66 | 91.32 | 92.32 | 93.76 | 88.23 | 93.82 | 92.32 | 92.98 | |
| Hydra Tensor ParallelInterconnect=NVLink, Number of GPUs=2, Input Tokens=1282026.04 | 89.53 | 91.13 | 40.52 | 91.13 | 88.32 | 48.63 | 33.65 | 89.33 | 47.65 | 88.35 | 43.65 | 75.59 | |
| Cinnamon Limb ParallelInterconnect=NVLink, Number of GPUs=2, Input Tokens=1282026.04 | 81.71 | 30.75 | 30.43 | 30.75 | 84.63 | 61.23 | 30.11 | 84.17 | 30.11 | 82.27 | 30.11 | 53.01 | |
| Plaintext Tensor ParallelInterconnect=NVLink, Number of GPUs=2, Input Tokens=1282026.04 | 49.52 | 50 | 50 | 50 | 49.87 | 50 | 49.83 | 50 | 50 | 50 | 49.83 | 49.85 | |
| HEBooster Limb ParallelInterconnect=NVLink, Number of GPUs=2, Input Tokens=1282026.04 | 31.71 | 30.75 | 30.43 | 30.75 | 34.63 | 31.5 | 30.11 | 34.17 | 30.11 | 31.27 | 30.11 | 31.86 |