Natural Language Understanding on SuperGLUE Zero-shot
88BoolQ AccuracyPaLM
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PaLMZero-shot=true, Parameters=540B2022.05 | 88 | 51.8 | 72.9 | 92.9 | 89.1 | 59.1 | 93 | 83.5 | 78.8 | |
| GLaM-MoEZero-shot=true, Parameters=1.2T2022.05 | 83 | 33.9 | 68.8 | 90.3 | 84.9 | 50.5 | 90 | 45.1 | 68.3 | |
| GLaM-DenseZero-shot=true, Parameters=8B2022.05 | 73.6 | 33.9 | 44 | 89.2 | 80.7 | 44 | 86 | 39 | 61.3 | |
| GLaM-MoEZero-shot=true, Experts=64E2022.05 | 72.2 | 40.7 | 60.3 | 88.9 | 81.8 | 49.5 | 86 | 52.4 | 66.5 | |
| PaLM-DenseZero-shot=true, Parameters=8B2022.05 | 68.3 | 41.1 | 54.2 | 87.8 | 78.9 | 47 | 86 | 47.5 | 63.9 | |
| GPT-3Zero-shot=true, Parameters=13B2022.05 | 66.2 | 19.6 | 62.8 | 89 | 64.4 | 0 | 84 | 71.4 | 57.2 | |
| UL2Zero-shot=true, Parameters=20B, Checkpoint=single ckpt2022.05 | 63.1 | 41.1 | 60.7 | 88.1 | 79.9 | 49.8 | 85 | 36.2 | 63 | |
| UL2Zero-shot=true, Parameters=20B, Checkpoint=best2022.05 | 63.1 | 50 | 60.7 | 88.1 | 80.6 | 55.2 | 88 | 36.2 | 65.2 | |
| GPT-3Zero-shot=true, Parameters=175B2022.05 | 60.5 | 46.4 | 63.5 | 90.2 | 65.4 | 0 | 91 | 72.9 | 61.2 | |
| T5-XXLZero-shot=true, Adaptation=LM adaptation2022.05 | 44.3 | 37.5 | 48.8 | 85.8 | 59.3 | 50.9 | 70 | 23 | 52.5 | |
| ST-MoE-32BZero-shot=true, Parameters=269B2022.05 | 40.8 | 41.1 | 52.7 | 50 | 57.5 | 50 | 56 | 30.3 | 47.6 |