Toxicity Classification on CivilComments sensitive attribute: MUSLIM (test)
59.9Balanced AccuracyT5TT-4-ERM
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| T5TT-4-ERMBase Model=T5, Trigger Tokens=4, Algorithm=ERM2022.12 | 59.9 | 0.15 | 0.17 | |
| ERMBackbone=T5, Trigger Tokens=42022.12 | 59.9 | 0.15 | 0.17 | |
| Fair-IJBackbone=BERT-NC, Constraint=Equality of Odds2022.12 | 59.8 | 0.126 | — | |
| Fair-IJBackbone=BERT-NC2022.12 | 59.8 | 0.126 | 0.008 | |
| ERMBackbone=BERT-NC, Constraint=Equality of Odds2022.12 | 59.3 | 0.326 | — | |
| ERMBackbone=BERT-NC, Constraint=Demographic Parity2022.12 | 59.3 | — | 0.261 | |
| T5TT-10-ERMBase Model=T5, Trigger Tokens=10, Algorithm=ERM2022.12 | 59.3 | 0.15 | 0.158 | |
| ERMBackbone=BERT-NC2022.12 | 59.3 | 0.326 | 0.261 | |
| ERMBackbone=T5, Trigger Tokens=102022.12 | 59.3 | 0.15 | 0.158 | |
| Gap RegularizationBackbone=BERT-NC, Constraint=Equality of Odds, Lambda=12022.12 | 59.1 | 0.144 | — | |
| T5TT-8-ERMBase Model=T5, Trigger Tokens=8, Algorithm=ERM2022.12 | 59.1 | 0.141 | 0.157 | |
| ERMBackbone=T5, Trigger Tokens=82022.12 | 59.1 | 0.141 | 0.157 | |
| GapRegBackbone=BERT-NC, λ=0.22022.12 | 59 | 0.166 | 0.136 | |
| HGRBackbone=BERT-NC, λ=0.22022.12 | 59 | 0.169 | 0.24 | |
| Fair-IJBackbone=BERT-LC, Constraint=Equality of Odds2022.12 | 58.6 | 0.125 | — | |
| Fair-IJBackbone=BERT-NC, Constraint=Demographic Parity2022.12 | 58.6 | — | 0.008 | |
| Fair-IJBackbone=BERT-LC2022.12 | 58.6 | 0.125 | 0.011 | |
| ERMBackbone=BERT-LC, Constraint=Demographic Parity2022.12 | 58.4 | — | 0.246 | |
| ERMBackbone=BERT-LC2022.12 | 58.4 | 0.314 | 0.246 | |
| ERMBackbone=BERT-TT-10, Constraint=Equality of Odds2022.12 | 58.3 | 0.348 | — | |
| ERMBackbone=BERT-TT-10, Constraint=Demographic Parity2022.12 | 58.3 | — | 0.234 | |
| GapRegBackbone=BERT-LC, λ=0.22022.12 | 58.3 | 0.198 | 0.149 | |
| ERMBackbone=BERT, Trigger Tokens=102022.12 | 58.3 | 0.348 | 0.234 | |
| Gap RegularizationBackbone=BERT-NC, Constraint=Demographic Parity, Lambda=12022.12 | 58.2 | — | 0.054 | |
| ERMBackbone=BERT-TT-8, Constraint=Equality of Odds2022.12 | 58.2 | 0.317 | — | |
| ERMBackbone=BERT-TT-8, Constraint=Demographic Parity2022.12 | 58.2 | — | 0.254 | |
| ERMBackbone=BERT, Trigger Tokens=82022.12 | 58.2 | 0.317 | 0.254 | |
| Gap RegularizationBackbone=BERT-LC, Constraint=Equality of Odds, Lambda=12022.12 | 57.9 | 0.133 | — | |
| GapRegBackbone=BERT-LC, λ=12022.12 | 57.9 | 0.144 | 0.071 | |
| GapRegBackbone=BERT-NC, λ=12022.12 | 57.9 | 0.144 | 0.054 | |
| Gap RegularizationBackbone=BERT-LC, Constraint=Demographic Parity, Lambda=12022.12 | 57.5 | — | 0.185 | |
| ERMBackbone=BERT-TT-4, Constraint=Equality of Odds2022.12 | 57.5 | 0.36 | — | |
| ERMBackbone=BERT-TT-4, Constraint=Demographic Parity2022.12 | 57.5 | — | 0.268 | |
| ERMBackbone=BERT, Trigger Tokens=42022.12 | 57.5 | 0.36 | 0.268 | |
| ERMBackbone=BERT-LC, Constraint=Equality of Odds2022.12 | 57.3 | 0.314 | — | |
| Fair-IJBackbone=BERT-TT-10, Constraint=Equality of Odds2022.12 | 57.2 | 0.11 | — | |
| Fair-IJBackbone=BERT, Trigger Tokens=102022.12 | 57.2 | 0.11 | 0.089 | |
| Fair-IJBackbone=BERT-LC, Constraint=Demographic Parity2022.12 | 57.1 | — | 0.011 | |
| HGRBackbone=BERT-LC, λ=0.22022.12 | 56.9 | 0.209 | 0.213 | |
| Fair-IJBackbone=BERT-TT-10, Constraint=Demographic Parity2022.12 | 56.6 | — | 0.089 | |
| Fair-IJBackbone=BERT-TT-8, Constraint=Equality of Odds2022.12 | 56.5 | 0.113 | — | |
| Fair-IJBackbone=BERT, Trigger Tokens=82022.12 | 56.5 | 0.113 | 0.071 | |
| Fair-IJBackbone=BERT-TT-8, Constraint=Demographic Parity2022.12 | 56.4 | — | 0.071 | |
| T5TT-8-Fair-IJBase Model=T5, Trigger Tokens=8, Algorithm=Fair-IJ, Fairness Metric Goal=ΔDP2022.12 | 56.4 | — | 0.002 | |
| Fair-IJBackbone=BERT-TT-4, Constraint=Equality of Odds2022.12 | 56 | 0.102 | — | |
| Fair-IJBackbone=BERT, Trigger Tokens=42022.12 | 56 | 0.102 | 0.042 | |
| Fair-IJBackbone=BERT-TT-4, Constraint=Demographic Parity2022.12 | 55.2 | — | 0.042 | |
| T5TT-4-Fair-IJBase Model=T5, Trigger Tokens=4, Algorithm=Fair-IJ, Fairness Metric Goal=ΔDP2022.12 | 55.2 | — | 0.008 | |
| T5TT-10-Fair-IJBase Model=T5, Trigger Tokens=10, Algorithm=Fair-IJ, Fairness Metric Goal=ΔDP2022.12 | 54.9 | — | 0.004 | |
| T5TT-10-Fair-IJBase Model=T5, Trigger Tokens=10, Algorithm=Fair-IJ, Fairness Metric Goal=ΔEO2022.12 | 54.5 | 0.027 | — | |
| Fair-IJBackbone=T5, Trigger Tokens=102022.12 | 54.5 | 0.027 | 0.004 | |
| HGRBackbone=BERT-NC, λ=12022.12 | 54 | 0.339 | 0.191 | |
| HGRBackbone=BERT-LC, λ=12022.12 | 53 | 0.372 | 0.149 | |
| T5TT-4-Fair-IJBase Model=T5, Trigger Tokens=4, Algorithm=Fair-IJ, Fairness Metric Goal=ΔEO2022.12 | 52.8 | 0.019 | — | |
| Fair-IJBackbone=T5, Trigger Tokens=42022.12 | 52.8 | 0.019 | 0.008 | |
| T5TT-8-Fair-IJBase Model=T5, Trigger Tokens=8, Algorithm=Fair-IJ, Fairness Metric Goal=ΔEO2022.12 | 52.6 | 0.019 | — | |
| Fair-IJBackbone=T5, Trigger Tokens=82022.12 | 52.6 | 0.019 | 0.002 |