Confidence Calibration on MultiNLI Mismatch (test)
0.0071ECEMIR
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MIRBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.0071 | 0.0107 | 0.0026 | 0.0258 | |
| IRBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.0088 | 0.0107 | 0.0036 | 0.0246 | |
| MIR + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0103 | 0.0136 | 0.004 | 0.0235 | |
| HB + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0105 | 0.015 | 0.0025 | 0.0251 | |
| HBBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.0111 | 0.0117 | 0.0036 | 0.0376 | |
| TS + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0122 | 0.0135 | 0.004 | 0.0278 | |
| ETS + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0124 | 0.0126 | 0.0039 | 0.0277 | |
| IR + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0143 | 0.0123 | 0.0034 | 0.0261 | |
| Conf + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0145 | 0.0146 | 0.0037 | 0.0278 | |
| ETSBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.0158 | 0.0156 | 0.0066 | 0.0304 | |
| TSBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.017 | 0.0176 | 0.0041 | 0.0303 | |
| PTS + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0242 | 0.0246 | 0.0093 | 0.0356 | |
| ConfBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.0247 | 0.0262 | 0.0085 | 0.0354 | |
| PTSK + PROCALBackbone=RoBERTa [24], Calibration Protocol=Ours2023.06 | 0.0426 | 0.044 | 0.0283 | 0.0515 | |
| PTSBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.0753 | 0.075 | 0.0421 | 0.0778 | |
| PTSKBackbone=RoBERTa [24], Calibration Protocol=Base2023.06 | 0.1014 | 0.1014 | 0.074 | 0.1033 |