Robust Classification on Civil Comments (test)
92.2Average AccuracyERM
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ERMGroup-oblivious setting=true2022.09 | 92.2 | 56 | — | — | |
| ERM2023.04 | 92.2 | 56.5 | — | — | |
| Focal LossGroup-oblivious setting=true2022.09 | 91.2 | 60.1 | — | — | |
| Sparse and Shared Feature Activations2023.04 | 91.2 | 75.45 | — | — | |
| JTTGroup Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 91.1 | 69.3 | — | — | |
| JTTGroup Info (Train/Val)=✗/★2026.02 | 91.1 | 69.3 | — | — | |
| LEIAGroup Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 90.9 | 72.9 | — | — | |
| LEIAGroup Info (Train/Val)=✗/★2026.02 | 90.9 | 72.9 | — | — | |
| JTTGroup-oblivious setting=true2022.09 | 90.7 | 67.4 | — | — | |
| UMIXGroup-oblivious setting=true2022.09 | 90.6 | 70.1 | — | — | |
| DRO2023.04 | 90.2 | 69 | — | — | |
| chi^2-DOROGroup-oblivious setting=true2022.09 | 90.1 | 63.8 | — | — | |
| CVaR-DOROGroup-oblivious setting=true2022.09 | 90 | 64.1 | — | — | |
| GICGroup Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 90 | 72.5 | — | — | |
| GICGroup Info (Train/Val)=✗/★2026.02 | 90 | 72.5 | — | — | |
| AFRGroup Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 89.8 | 68.7 | — | — | |
| AFRGroup Info (Train/Val)=✗/★2026.02 | 89.8 | 68.7 | — | — | |
| chi^2-DROGroup-oblivious setting=true2022.09 | 89.4 | 64.2 | — | — | |
| LISAGroup Info (Train/Val)=✓/★, Group labels usage in training=used2026.02 | 89.2 | 72.6 | — | — | |
| CVaR-DROGroup-oblivious setting=true2022.09 | 89.1 | 62.3 | — | — | |
| ERMValidation selection=minimax2026.01 | 89.1 | 2.2 | 0.869 | — | |
| GALAXYinitial_labels=100, query_schedule=10 x 102025.10 | 87.77 | — | — | — | |
| DFRGroup Info (Train/Val)=✗/✓✓, Group labels usage in training=used2026.02 | 87.2 | 70.1 | — | — | |
| Confidenceinitial_labels=100, query_schedule=10 x 102025.10 | 86.43 | — | — | — | |
| ParBaLS-MAP EPIGinitial_labels=100, query_schedule=10 x 102025.10 | 85.67 | — | — | — | |
| ParBaLS EPIGinitial_labels=100, query_schedule=10 x 102025.10 | 85.59 | — | — | — | |
| IRM*Group Info (Train/Val)=✓/★, Group labels usage in training=used2026.02 | 85.5 | 63.2 | — | — | |
| ERM*Group Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 85.4 | 63.2 | — | — | |
| ERMGroup Info (Train/Val)=✗/★2026.02 | 85.4 | 63.2 | — | — | |
| CVaRDRO*Group Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 83.5 | 68.7 | — | — | |
| CVaRDROGroup Info (Train/Val)=✗/★2026.02 | 83.5 | 68.7 | — | — | |
| CVaRValidation selection=minimax2026.01 | 83.5 | 23 | 0.605 | — | |
| ReWeightCRT*Group Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 83.4 | 68.2 | — | — | |
| ReWeightCRTGroup Info (Train/Val)=✗/★2026.02 | 83.4 | 68.2 | — | — | |
| CRT*Group Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 83 | 68.5 | — | — | |
| CRTGroup Info (Train/Val)=✗/★2026.02 | 83 | 68.5 | — | — | |
| LV-GroupValidation selection=minimax2026.01 | 82.8 | 51.6 | 0.312 | — | |
| CnCGroup Info (Train/Val)=✗/★, Group labels usage in training=not used2026.02 | 81.7 | 68.9 | — | — | |
| CnCGroup Info (Train/Val)=✗/★2026.02 | 81.7 | 68.9 | — | — | |
| OTTERBackbone=BERT, Setting=Zero-shot2024.04 | 81.4 | — | — | — | |
| Group DROGroup Info (Train/Val)=✓/★, Group labels usage in training=used2026.02 | 81.4 | 77.7 | — | — | |
| Randominitial_labels=100, query_schedule=10 x 102025.10 | 80.88 | — | — | — | |
| GLISTERinitial_labels=100, query_schedule=10 x 102025.10 | 80.19 | — | — | — | |
| χ2-DROValidation selection=minimax2026.01 | 79.8 | 24.7 | 0.551 | — | |
| BADGEinitial_labels=100, query_schedule=10 x 102025.10 | 79.76 | — | — | — | |
| GroupDROValidation selection=minimax2026.01 | 77 | 45.6 | 0.314 | — | |
| GroupDRO-BValidation selection=minimax, Training data=bulk-filtered2026.01 | 76.8 | 45.3 | 0.315 | — | |
| CoreSetinitial_labels=100, query_schedule=10 x 102025.10 | 72.33 | — | — | — | |
| DPEGroup Info (Train/Val)=✗/✓, Group labels usage in training=used2026.02 | 71.5 | 82.2 | — | — | |
| Prior MatchingBackbone=BERT, Setting=Zero-shot2024.04 | 57.2 | — | — | — | |
| Zero-shotBackbone=BERT, Setting=Zero-shot2024.04 | 48.4 | — | — | — | |
| CAMLAcquisition Function=Random, Backbone=BERT2026.05 | — | — | — | 49.2 | |
| CAMLAcquisition Function=Confidence, Backbone=BERT2026.05 | — | — | — | 58 | |
| CAMLAcquisition Function=CoreSet, Backbone=BERT2026.05 | — | — | — | 63.1 | |
| CAMLAcquisition Function=BADGE, Backbone=BERT2026.05 | — | — | — | 62.7 | |
| CB last-layer retrainingAnnotations Group=false, Annotations Class=true2023.09 | — | 80.4 | — | — | |
| Class-balanced ERMAnnotations Group=false, Annotations Class=false2023.09 | — | 61.4 | — | — | |
| DFRAnnotations Group=true, Annotations Class=true2023.09 | — | 78.8 | — | — | |
| DFR (our impl.)Annotations Group=true, Annotations Class=true2023.09 | — | 81.8 | — | — | |
| ES disagreement SELFAnnotations Group=false, Annotations Class=false2023.09 | — | 79.1 | — | — | |
| Group DRO-ESAnnotations Group=true, Annotations Class=true2023.09 | — | 80.4 | — | — | |
| RWY-ESAnnotations Group=false, Annotations Class=true2023.09 | — | 78.9 | — | — | |
| ShuffleAcquisition Function=Random, Backbone=BERT2026.05 | — | — | — | 40.1 | |
| ShuffleAcquisition Function=Confidence, Backbone=BERT2026.05 | — | — | — | 40.3 | |
| ShuffleAcquisition Function=CoreSet, Backbone=BERT2026.05 | — | — | — | 39.1 | |
| ShuffleAcquisition Function=BADGE, Backbone=BERT2026.05 | — | — | — | 40.3 |