List your company
ResearchFormerly WizWand

Question Answering on SQuAD v2.0 (val)

83.3F1Full Precision
44.354.42564.5574.675Sep 27, 2022
Updated 8mo ago

Evaluation results

MethodLinks
Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=RoBERTa2022.0983.380.26—
Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=RoBERTa2022.0982.9479.72—
OMSEBits (W-E-A)=8-8-8, Model Architecture=RoBERTa2022.0982.5379.41—
Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=BART2022.0980.8277.41—
Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=BART2022.0980.5577.04—
OMSEBits (W-E-A)=8-8-8, Model Architecture=BART2022.0979.6276.12—
Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=BERT2022.0977.3473.6—
Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=BERT2022.0976.9373.14—
OMSEBits (W-E-A)=8-8-8, Model Architecture=BERT2022.0976.8873.08—
Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=BERT2022.0974.6970.55—
Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=BART2022.0974.4470.36—
EasyQuantBits (W-E-A)=6-6-6, Model Architecture=BERT2022.0971.9568.06—
EasyQuantBits (W-E-A)=6-6-6, Model Architecture=BART2022.0969.9364.94—
PercentileBits (W-E-A)=6-6-6, Model Architecture=BERT2022.0969.1265.64—
Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.0968.4764.1—
PercentileBits (W-E-A)=6-6-6, Model Architecture=BART2022.0968.4463.29—
OMSEBits (W-E-A)=6-6-6, Model Architecture=BART2022.0967.8963.29—
OMSEBits (W-E-A)=6-6-6, Model Architecture=BERT2022.0967.5263.09—
PercentileBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.0956.3851.58—
EasyQuantBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.0947.9942.21—
OMSEBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.0945.839.95—
BERT-basemodel=BERT-base2026.02——18.6
Deep Ensembleensemble_size=52026.02——16.8
MC Dropoutp=0.1, type=uniform2026.02——17.3
MC Dropouttype=component-specific2026.02——18.6
SNGP2026.02——0.1
Temp. Scalingpost-hoc=true2026.02——5.7
UAT-LITE2026.02——16.3
VI-LastLayer2026.02——11.9