Natural Language Understanding on GLUE (val) (CoLA, MNLI, MRPC, QNLI, RTE)
74.6RTE AccuracySmoothSMoE
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SmoothSMoEBackbone=BERT-large, K (total experts)=16, k (active experts)=42026.06 | 74.6 | 89.88 | 64.82 | 92.53 | 86.82 | 81.73 | |
| SMoEBackbone=BERT-large, K (total experts)=16, k (active experts)=42026.06 | 73.85 | 89.26 | 63.9 | 92.2 | 86.49 | 81.14 | |
| SmoothSMoEBackbone=BERT-large, K (total experts)=16, k (active experts)=22026.06 | 73.4 | 90.15 | 65.41 | 92.4 | 86.9 | 81.65 | |
| SMoEBackbone=BERT-large, K (total experts)=16, k (active experts)=22026.06 | 73.28 | 89.17 | 64.25 | 92.56 | 86.6 | 81.17 | |
| ReMoEBackbone=BERT-large, K (total experts)=16, k (active experts)=22026.06 | 73.1 | 88.6 | 64.9 | 92.53 | 86.69 | 81.18 | |
| ReMoEBackbone=BERT-large, K (total experts)=16, k (active experts)=42026.06 | 72.2 | 89.49 | 65.07 | 92.51 | 86.51 | 81.16 |