Natural Language Inference on MultiNLI (test) (Matched & Mismatched)
88.05Average Worst-Group AccuracyNaive + P
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Naive + PContext size=642024.10 | 88.05 | — | — | — | — | — | — | 0.38 | |
| Naive + PContext size=2562024.10 | 87.99 | — | — | — | — | — | — | 0.2 | |
| Proposed + G + PContext size=642024.10 | 87.97 | — | — | — | — | — | — | 0.3 | |
| Proposed + PContext size=642024.10 | 87.86 | — | — | — | — | — | — | 0.62 | |
| Proposed + G + PContext size=162024.10 | 87.81 | — | — | — | — | — | — | 0.56 | |
| Proposed + G + PContext size=2562024.10 | 87.78 | — | — | — | — | — | — | 0.31 | |
| Proposed + PContext size=162024.10 | 87.75 | — | — | — | — | — | — | 0.24 | |
| Proposed + G + PContext size=322024.10 | 87.74 | — | — | — | — | — | — | 0.5 | |
| Proposed + PContext size=5122024.10 | 87.73 | — | — | — | — | — | — | 0.3 | |
| Proposed + PContext size=322024.10 | 87.7 | — | — | — | — | — | — | 0.43 | |
| Proposed + GContext size=2562024.10 | 87.69 | — | — | — | — | — | — | 0.51 | |
| Proposed + G + PContext size=5122024.10 | 87.68 | — | — | — | — | — | — | 0.12 | |
| Proposed + G + PContext size=82024.10 | 87.57 | — | — | — | — | — | — | 0.34 | |
| Naive + PContext size=162024.10 | 87.56 | — | — | — | — | — | — | 0.77 | |
| Proposed + PContext size=2562024.10 | 87.56 | — | — | — | — | — | — | 0.46 | |
| Proposed + G + PContext size=1282024.10 | 87.56 | — | — | — | — | — | — | 0.27 | |
| ERMContext size=5122024.10 | 87.55 | — | — | — | — | — | — | 0.41 | |
| Naive + PContext size=5122024.10 | 87.54 | — | — | — | — | — | — | 0.34 | |
| ProposedContext size=162024.10 | 87.54 | — | — | — | — | — | — | 0.42 | |
| Naive + PContext size=322024.10 | 87.5 | — | — | — | — | — | — | 0.57 | |
| Proposed + GContext size=5122024.10 | 87.5 | — | — | — | — | — | — | 0.55 | |
| ProposedContext size=82024.10 | 87.45 | — | — | — | — | — | — | 0.58 | |
| Naive + PContext size=82024.10 | 87.43 | — | — | — | — | — | — | 1.05 | |
| Proposed + PContext size=82024.10 | 87.43 | — | — | — | — | — | — | 0.48 | |
| Proposed + GContext size=162024.10 | 87.37 | — | — | — | — | — | — | 0.35 | |
| Naive + PContext size=1282024.10 | 87.35 | — | — | — | — | — | — | 0.27 | |
| ProposedContext size=2562024.10 | 87.35 | — | — | — | — | — | — | 0.19 | |
| Proposed + G + PContext size=42024.10 | 87.32 | — | — | — | — | — | — | 0.76 | |
| Proposed + PContext size=1282024.10 | 87.31 | — | — | — | — | — | — | 0.53 | |
| ERMContext size=642024.10 | 87.3 | — | — | — | — | — | — | 0.78 | |
| Naive + PContext size=42024.10 | 87.29 | — | — | — | — | — | — | 0.95 | |
| Proposed + GContext size=82024.10 | 87.23 | — | — | — | — | — | — | 0.42 | |
| GroupDROContext size=5122024.10 | 87.23 | — | — | — | — | — | — | 0.49 | |
| Proposed + PContext size=42024.10 | 87.22 | — | — | — | — | — | — | 0.55 | |
| GroupDROContext size=642024.10 | 87.16 | — | — | — | — | — | — | 0.55 | |
| Proposed + GContext size=22024.10 | 87.15 | — | — | — | — | — | — | 0.82 | |
| GroupDROContext size=1282024.10 | 87.14 | — | — | — | — | — | — | 0.37 | |
| ERMContext size=2562024.10 | 87.1 | — | — | — | — | — | — | 0.48 | |
| GroupDROContext size=2562024.10 | 87.1 | — | — | — | — | — | — | 0.14 | |
| ERMContext size=1282024.10 | 87.04 | — | — | — | — | — | — | 0.55 | |
| Proposed + GContext size=322024.10 | 87.02 | — | — | — | — | — | — | 0.27 | |
| ProposedContext size=322024.10 | 86.99 | — | — | — | — | — | — | 0.26 | |
| ProposedContext size=5122024.10 | 86.89 | — | — | — | — | — | — | 0.3 | |
| ERMContext size=162024.10 | 86.82 | — | — | — | — | — | — | 0.54 | |
| ERMContext size=322024.10 | 86.76 | — | — | — | — | — | — | 0.54 | |
| GroupDROContext size=162024.10 | 86.75 | — | — | — | — | — | — | 0.47 | |
| ProposedContext size=22024.10 | 86.62 | — | — | — | — | — | — | 0.65 | |
| GroupDROContext size=322024.10 | 86.55 | — | — | — | — | — | — | 0.3 | |
| 1-NNContext size=5122024.10 | 86.53 | — | — | — | — | — | — | 0.29 | |
| ProposedContext size=642024.10 | 86.51 | — | — | — | — | — | — | 0.27 | |
| Proposed + GContext size=642024.10 | 86.51 | — | — | — | — | — | — | 0.58 | |
| Proposed + GContext size=1282024.10 | 86.47 | — | — | — | — | — | — | 0.49 | |
| 1-NNContext size=1282024.10 | 86.44 | — | — | — | — | — | — | 0.81 | |
| 1-NNContext size=162024.10 | 86.39 | — | — | — | — | — | — | 0.76 | |
| Proposed + GContext size=42024.10 | 86.23 | — | — | — | — | — | — | 0.65 | |
| ProposedContext size=1282024.10 | 86.11 | — | — | — | — | — | — | 0.29 | |
| 1-NNContext size=642024.10 | 86.04 | — | — | — | — | — | — | 0.93 | |
| ProposedContext size=42024.10 | 86.02 | — | — | — | — | — | — | 0.33 | |
| 1-NNContext size=322024.10 | 86.01 | — | — | — | — | — | — | 0.4 | |
| 1-NNContext size=2562024.10 | 85.9 | — | — | — | — | — | — | 0.43 | |
| Proposed + PContext size=22024.10 | 85.84 | — | — | — | — | — | — | 1.12 | |
| 1-NNContext size=82024.10 | 85.84 | — | — | — | — | — | — | 0.84 | |
| GroupDROContext size=82024.10 | 85.45 | — | — | — | — | — | — | 1.1 | |
| Proposed + G + PContext size=22024.10 | 85.41 | — | — | — | — | — | — | 1.38 | |
| Naive + PContext size=22024.10 | 85.3 | — | — | — | — | — | — | 1.58 | |
| NaiveContext size=642024.10 | 85.2 | — | — | — | — | — | — | 0.24 | |
| ERMContext size=82024.10 | 85.1 | — | — | — | — | — | — | 1.13 | |
| NaiveContext size=82024.10 | 84.74 | — | — | — | — | — | — | 0.37 | |
| NaiveContext size=42024.10 | 84.58 | — | — | — | — | — | — | 0.37 | |
| NaiveContext size=1282024.10 | 84.38 | — | — | — | — | — | — | 0.32 | |
| NaiveContext size=2562024.10 | 84.3 | — | — | — | — | — | — | 0.33 | |
| NaiveContext size=22024.10 | 84.15 | — | — | — | — | — | — | 0.3 | |
| NaiveContext size=162024.10 | 84.13 | — | — | — | — | — | — | 0.45 | |
| NaiveContext size=5122024.10 | 84.05 | — | — | — | — | — | — | 0.39 | |
| NaiveContext size=322024.10 | 83.92 | — | — | — | — | — | — | 0.25 | |
| 1-NNContext size=42024.10 | 82.41 | — | — | — | — | — | — | 0.81 | |
| ERMContext size=42024.10 | 81.71 | — | — | — | — | — | — | 0.58 | |
| GroupDROContext size=42024.10 | 81.56 | — | — | — | — | — | — | 0.75 | |
| GroupDROContext size=22024.10 | 76.54 | — | — | — | — | — | — | 0.42 | |
| ERMContext size=22024.10 | 76.4 | — | — | — | — | — | — | 0.65 | |
| 1-NNContext size=22024.10 | 64.49 | — | — | — | — | — | — | 0.9 | |
| AugmentationMitigation Approach=Augmentation, Base Model=ELECTRA-small2026.04 | — | — | — | 81.78 | — | — | — | — | |
| AugmentationMitigation Approach=Augmentation, Base Model=ELECTRA-small2026.04 | — | — | — | 81 | — | — | — | — | |
| BERT2018.04 | — | 86.7 | 85.9 | — | — | — | — | — | |
| CAFEEmbedding Dimension=300D, Model Architecture Class=Integrated neural network2019.07 | — | 78.7 | 77.9 | — | — | — | — | — | |
| CAFEEmbedding Dimension=300D, Model Architecture Class=Ensemble2019.07 | — | 80.2 | 79 | — | — | — | — | — | |
| CB last-layer retrainingAnnotations Group=false, Annotations Class=true2023.09 | — | — | — | — | — | 64.7 | — | — | |
| Class-balanced ERMAnnotations Group=false, Annotations Class=false2023.09 | — | — | — | — | — | 69.2 | — | — | |
| CRTGroup Info (Train/Val)=✗/✗2026.02 | — | — | — | — | — | 65.4 | 80.2 | — | |
| CVaR DROGroup annotated in train?=false, Group annotated in val?=true2023.05 | — | — | — | 82 | 68 | — | — | — | |
| CVaR DROGroup annotated in train?=false, Group annotated in val?=false2023.05 | — | — | — | 81.8 | 61.8 | — | — | — | |
| CVaRDROGroup Info (Train/Val)=✗/✗2026.02 | — | — | — | — | — | 48.2 | 75.4 | — | |
| DFRAnnotations Group=true, Annotations Class=true2023.09 | — | — | — | — | — | 72.6 | — | — | |
| DFRGroup Info (Train/Val)=✗/✗2026.02 | — | — | — | — | — | 63.8 | 80.2 | — | |
| DFR2026.04 | — | — | — | 82.4 | — | 57.1 | — | — | |
| DFR (our impl.)Annotations Group=true, Annotations Class=true2023.09 | — | — | — | — | — | 70.8 | — | — | |
| DIIN2018.04 | — | 78.8 | 77.8 | — | — | — | — | — | |
| DIINModel Architecture Class=Integrated neural network2019.07 | — | 78.8 | 77.8 | — | — | — | — | — | |
| DIINModel Architecture Class=Ensemble2019.07 | — | 80 | 78.7 | — | — | — | — | — | |
| DMANModel Architecture Class=Integrated neural network2019.07 | — | 78.9 | 78.2 | — | — | — | — | — |