GPU Utilisation on GLUE fine-tuning base
96GPU UtilisationMoE-MPMC
Evaluation results
| Method | Links | |
|---|---|---|
| MoE-MPMCModel Architecture=switch-base, Number of Experts=82026.05 | 96 | |
| SiDA-MoEModel Architecture=switch-base, Number of Experts=82026.05 | 89 | |
| MoE-MPMCModel Architecture=switch-base, Number of Experts=642026.05 | 89 | |
| MoE-MPMCModel Architecture=switch-base, Number of Experts=1282026.05 | 82 | |
| SiDA-MoEModel Architecture=switch-base, Number of Experts=642026.05 | 79 | |
| SiDA-MoEModel Architecture=switch-base, Number of Experts=1282026.05 | 63 | |
| SwitchTransformers (Baseline)Model Architecture=switch-base, Number of Experts=82026.05 | 7.5 | |
| SwitchTransformers (Baseline)Model Architecture=switch-base, Number of Experts=642026.05 | 3.4 | |
| SwitchTransformers (Baseline)Model Architecture=switch-base, Number of Experts=1282026.05 | 1.3 |