Natural Language Understanding on GLUE (val) - Subset (SST-2, MNLI, QNLI)
87.5SST-2 AccuracyGQA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| GQAModel size=60M, Pre-training dataset=WikiText-103, Evaluation=fine-tuned, Number of seeds=32026.04 | 87.5 | 72.9 | 81.2 | |
| StandardModel size=60M, Pre-training dataset=WikiText-103, Evaluation=fine-tuned, Number of seeds=32026.04 | 87.1 | 72.9 | 81.8 | |
| Coupled QKModel size=60M, Pre-training dataset=WikiText-103, Evaluation=fine-tuned, Number of seeds=32026.04 | 86.8 | 73.1 | 81.7 | |
| Diff AttentionModel size=60M, Pre-training dataset=WikiText-103, Evaluation=fine-tuned, Number of seeds=32026.04 | 86.4 | 73.7 | 82.2 |