Natural Language Understanding on GLUE and SuperGLUE (dev)
89.5BoolQGShard (GS Top-2)
Evaluation Results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GShard (GS Top-2)Expert Size=8B, Num Experts=64, Gating=Top-2, Protocol=Fine-tuning2022.02 | 89.5 | 96.7 | 87.5 | 91.4 | 91.7 | 94.9 | 92.5 | 92.2 | 98 | 76.4 | 82.8 | 90.3 | |
| Expert Choice (EC-CF2)Expert Size=8B, Num Experts=64, Gating=Expert Choice (CF=2), Protocol=Fine-tuning2022.02 | 89.2 | 100 | 89.1 | 91.1 | 90.6 | 95 | 93.8 | 95.2 | 97.7 | 83.8 | 92.8 | 92.6 | |
| Switch Transformer (ST Top-1)Expert Size=8B, Num Experts=64, Gating=Top-1, Protocol=Fine-tuning2022.02 | 89.1 | 93.8 | 88.3 | 90.7 | 89.3 | 94.5 | 92.1 | 91 | 97.1 | 74.5 | 78.1 | 88.9 | |
| Expert Choice (EC-CF2)Expert Size=100M, Num Experts=64, Gating=Expert Choice (CF=2), Protocol=Fine-tuning2022.02 | 79.7 | 89.1 | 88.3 | 86.7 | 84.4 | 91.3 | 91 | 81.6 | 95.1 | 65.6 | 71.7 | 84 | |
| Expert Choice (EC-CF2)Expert Size=100M, Num Experts=32, Gating=Expert Choice (CF=2), Protocol=Fine-tuning2022.02 | 79.3 | 92.2 | 93.8 | 88 | 84.4 | 92.5 | 92 | 78.1 | 95.4 | 69.8 | 68.8 | 85 | |
| GShard (GS Top-2)Expert Size=100M, Num Experts=32, Gating=Top-2, Protocol=Fine-tuning2022.02 | 79 | 81.3 | 92.2 | 87.8 | 85.2 | 91.9 | 91.5 | 79.1 | 94.4 | 65.9 | 64.1 | 83.5 | |
| GShard (GS Top-2)Expert Size=100M, Num Experts=64, Gating=Top-2, Protocol=Fine-tuning2022.02 | 77.5 | 84.4 | 85.2 | 85.2 | 81.3 | 89.7 | 90.5 | 79.3 | 95.1 | 67.8 | 68.8 | 82.2 | |
| Switch Transformer (ST Top-1)Expert Size=100M, Num Experts=128, Gating=Top-1, Protocol=Fine-tuning2022.02 | 77.4 | 87.5 | 78.9 | 82.3 | 82.6 | 89.5 | 90.6 | 77 | 92 | 67.8 | 65.6 | 81 | |
| Expert Choice (EC-CF2)Expert Size=100M, Num Experts=128, Gating=Expert Choice (CF=2), Protocol=Fine-tuning2022.02 | 76.9 | 89.1 | 86.7 | 84.9 | 83.1 | 89 | 90.4 | 78.5 | 94.6 | 68.1 | 67.2 | 82.6 | |
| GShard (GS Top-2)Expert Size=100M, Num Experts=128, Gating=Top-2, Protocol=Fine-tuning2022.02 | 76.5 | 80.9 | 84 | 83.6 | 81 | 88.6 | 90.3 | 78.9 | 94.5 | 65.5 | 70.3 | 81.3 | |
| Switch Transformer (ST Top-1)Expert Size=100M, Num Experts=32, Gating=Top-1, Protocol=Fine-tuning2022.02 | 74.5 | 80.6 | 87.5 | 83.1 | 82.3 | 91.6 | 90.1 | 75 | 93.3 | 62.5 | 65.6 | 80.6 | |
| Switch Transformer (ST Top-1)Expert Size=100M, Num Experts=64, Gating=Top-1, Protocol=Fine-tuning2022.02 | 73.2 | 85.9 | 64.1 | 80.8 | 81.3 | 89.4 | 88.9 | 74.1 | 91.8 | 64.4 | 68.8 | 78.4 |