Natural Language Understanding on GLUE (7-Task Subset: QQP, MNLI-m, SST-2, QNLI, RTE, MRPC, STS-B)
84.7MNLI-m AccuracyBERTbase
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| BERTbaseSize=418M2026.06 | 84.7 | 91.3 | 93.3 | 91.7 | 72.6 | 88.2 | 89.4 | 87.31 | |
| TinyBERT6Size=207M2026.06 | 84.6 | — | 93.1 | 90.4 | 70 | 87.3 | 83.7 | 84.85 | |
| DistilBERTSize=207M2026.06 | 82.2 | 88.5 | 91.3 | 89.2 | 59.9 | 87.5 | 86.9 | 83.64 | |
| Otters++Size=13.4M, Evaluation Mode=Variation, KV quantization=1 bit2026.06 | 79.93 | 88.24 | 90.53 | 88.19 | 71.28 | 86.09 | 84.93 | 84.17 | |
| Otters++Size=13.4M, Evaluation Mode=Nominal, KV quantization=1 bit2026.06 | 79.56 | 88.03 | 91.97 | 87.88 | 68.95 | 87 | 85.79 | 84.17 | |
| OttersSize=13.4M2026.06 | 78.5 | 87.67 | 91.28 | 86.42 | 68.95 | 84.56 | 85.19 | 83.22 | |
| SpikingBERTSize=50M2026.06 | 78.1 | 86.8 | 88.2 | 85.2 | 66.1 | 79.2 | 82.2 | 80.83 | |
| 1-bit SorbetSize=13.4M2026.06 | 77.3 | 86.5 | 90.4 | 86.1 | 60.3 | 79.9 | 78.1 | 79.8 | |
| BiTSize=13.4M2026.06 | 77.1 | 82.9 | 87.7 | 85.7 | 58.8 | 79.7 | 71.1 | 77.57 | |
| SpikeLMSize=*2026.06 | 76 | 87.9 | 86.5 | 84.9 | 65.3 | 78.7 | 84.3 | 80.51 | |
| 1-bit SpikeLMSize=*2026.06 | 74.9 | 87.2 | 86.6 | 84.5 | 65.7 | 78.9 | 83.9 | 80.24 | |
| SpikingFormerSize=*2026.06 | 67.8 | 83.8 | 82.7 | 74.6 | 58.8 | 74 | 72.3 | 73.43 | |
| Q2BERTSize=43.0M2026.06 | 47.2 | 67 | 80.6 | 61.3 | 52.7 | 68.4 | 4.4 | 54.51 |