Question Answering on SQuAD v2.0 (val)
83.3F1Full Precision
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=RoBERTa2022.09 | 83.3 | 80.26 | — | |
| Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=RoBERTa2022.09 | 82.94 | 79.72 | — | |
| OMSEBits (W-E-A)=8-8-8, Model Architecture=RoBERTa2022.09 | 82.53 | 79.41 | — | |
| Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=BART2022.09 | 80.82 | 77.41 | — | |
| Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=BART2022.09 | 80.55 | 77.04 | — | |
| OMSEBits (W-E-A)=8-8-8, Model Architecture=BART2022.09 | 79.62 | 76.12 | — | |
| Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=BERT2022.09 | 77.34 | 73.6 | — | |
| Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=BERT2022.09 | 76.93 | 73.14 | — | |
| OMSEBits (W-E-A)=8-8-8, Model Architecture=BERT2022.09 | 76.88 | 73.08 | — | |
| Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 74.69 | 70.55 | — | |
| Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 74.44 | 70.36 | — | |
| EasyQuantBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 71.95 | 68.06 | — | |
| EasyQuantBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 69.93 | 64.94 | — | |
| PercentileBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 69.12 | 65.64 | — | |
| Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 68.47 | 64.1 | — | |
| PercentileBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 68.44 | 63.29 | — | |
| OMSEBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 67.89 | 63.29 | — | |
| OMSEBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 67.52 | 63.09 | — | |
| PercentileBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 56.38 | 51.58 | — | |
| EasyQuantBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 47.99 | 42.21 | — | |
| OMSEBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 45.8 | 39.95 | — | |
| BERT-basemodel=BERT-base2026.02 | — | — | 18.6 | |
| Deep Ensembleensemble_size=52026.02 | — | — | 16.8 | |
| MC Dropoutp=0.1, type=uniform2026.02 | — | — | 17.3 | |
| MC Dropouttype=component-specific2026.02 | — | — | 18.6 | |
| SNGP2026.02 | — | — | 0.1 | |
| Temp. Scalingpost-hoc=true2026.02 | — | — | 5.7 | |
| UAT-LITE2026.02 | — | — | 16.3 | |
| VI-LastLayer2026.02 | — | — | 11.9 |