Natural Language Understanding on GLUE (test) (Subset: QNLI, CoLA, STS-B, MRPC, RTE)
92.4QNLI AccuracyPlain-text
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Plain-textModels=BERT_LARGE, GeLU Approx.=GeLU, Softmax Approx.=Softmax2024.01 | 92.4 | 61.7 | 90.2 | 90.6 | 75.5 | 82.1 | |
| PUMAModels=BERT_LARGE, GeLU Approx.=GeLU, Softmax Approx.=Softmax2024.01 | 92.4 | 61.7 | 90.2 | 90.6 | 75.5 | 82.1 | |
| SecFormerModels=BERT_LARGE, GeLU Approx.=GeLU, Softmax Approx.=2Quad, Distillation=true2024.01 | 92 | 61.3 | 89.2 | 88.7 | 72.6 | 80.8 | |
| Plain-textModels=BERT_BASE, GeLU Approx.=GeLU, Softmax Approx.=Softmax2024.01 | 91.7 | 57.8 | 89.1 | 90.3 | 69.7 | 79.7 | |
| PUMAModels=BERT_BASE, GeLU Approx.=GeLU, Softmax Approx.=Softmax2024.01 | 91.7 | 57.8 | 89.1 | 90.3 | 69.7 | 79.7 | |
| SecFormerModels=BERT_BASE, GeLU Approx.=GeLU, Softmax Approx.=2Quad, Distillation=true2024.01 | 91.2 | 57.1 | 87.4 | 89.2 | 69 | 78.8 | |
| SecFormerModels=BERT_LARGE, GeLU Approx.=GeLU, Softmax Approx.=2Quad, Distillation=false2024.01 | 90.8 | 60.8 | 89 | 87.6 | 69.7 | 79.6 | |
| MPCFormerModels=BERT_BASE, GeLU Approx.=Quad, Softmax Approx.=2Quad, Distillation=true2024.01 | 90.6 | 52.6 | 80.3 | 88.7 | 64.9 | 75.4 | |
| SecFormerModels=BERT_BASE, GeLU Approx.=GeLU, Softmax Approx.=2Quad, Distillation=false2024.01 | 89.3 | 57 | 86.2 | 83.8 | 63.2 | 75.9 | |
| MPCFormerModels=BERT_LARGE, GeLU Approx.=Quad, Softmax Approx.=2Quad, Distillation=true2024.01 | 87.8 | 0 | 52.1 | 81.4 | 59.2 | 56.1 | |
| MPCFormerModels=BERT_BASE, GeLU Approx.=Quad, Softmax Approx.=2Quad, Distillation=false2024.01 | 69.8 | 0 | 36.1 | 81.2 | 52.7 | 48 | |
| MPCFormerModels=BERT_LARGE, GeLU Approx.=Quad, Softmax Approx.=2Quad, Distillation=false2024.01 | 49.5 | 0 | 0 | 81.2 | 52.7 | 36.7 |