Sentiment Analysis on IMDB (test) (Adversarial Robustness)
94.33Clean Accuracy (%)RanMASK
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RanMASKBackbone=RoBERTa-base2023.10 | 94.33 | 43.29 | 54.1 | 2,951.45 | 53.67 | 44.2 | 2,706.83 | 47.45 | 49.4 | 3,611.03 | — | — | — | 1.1 | — | — | — | — | |
| FreeLBBackbone=RoBERTa-base2023.10 | 94.2 | 68.3 | 29.8 | 726.84 | 75.19 | 23.1 | 647.98 | 72.95 | 25.4 | 887.31 | — | — | — | 0.97 | — | — | — | — | |
| InfoBERTBackbone=RoBERTa-base2023.10 | 94.18 | 70.61 | 27.6 | 583.97 | 83.53 | 15.5 | 518.17 | 78.14 | 20.9 | 684.64 | — | — | — | 0.95 | — | — | — | — | |
| SAFERBackbone=RoBERTa-base2023.10 | 93.75 | 32.27 | 63.6 | 1,874.79 | 41.77 | 54.1 | 1,415.56 | 34.21 | 62.3 | 2,059.37 | — | — | — | 0.52 | — | — | — | — | |
| PRO-BERTnorm=MCP, Combined with=AT2024.10 | 93.6 | 42 | 56.1 | — | — | — | — | 55.3 | 41 | — | — | — | — | — | 60.8 | 39 | 61 | 37.6 | |
| PRO-BERTnorm=MCP2024.10 | 93.5 | 22.1 | 76.9 | — | — | — | — | 44.6 | 53.2 | — | — | — | — | — | 55.5 | 41.8 | 56.3 | 41.1 | |
| Flooding-Xtraining_type=adversarial training2023.06 | 93.4 | 45.5 | 53.5 | 2,015.4 | 37.3 | 60.8 | 2,448.7 | 62.3 | 35.8 | 1,187.9 | — | — | — | — | — | — | — | — | |
| DSRM2023.06 | 93.4 | 56.3 | 39 | 2,215.3 | 54.1 | 41.2 | 3,309.8 | 67.2 | 28.9 | 1,207.7 | — | — | — | — | — | — | — | — | |
| InfoBERTmethod_category=SOTA defense2023.06 | 93.3 | 49.6 | 49.1 | 1,932.3 | 47.2 | 51.3 | 3,088.8 | 53.8 | 44.7 | 1,070.4 | — | — | — | — | — | — | — | — | |
| ROBERTA2024.10 | 93.3 | 23.7 | 74.6 | — | — | — | — | 9.4 | 89.9 | — | — | — | — | — | 36.5 | 60.9 | 19.5 | 79.1 | |
| PRO-BERTnorm=l12024.10 | 93.3 | 24.6 | 73.6 | — | — | — | — | 13 | 86.1 | — | — | — | — | — | 36 | 61.4 | 32.7 | 65 | |
| RoBERTa-baseBackbone=RoBERTa-base2023.10 | 93.23 | 87.79 | 11.4 | 456.35 | 91.29 | 8.2 | 439.79 | 92.63 | 6.9 | 517.58 | — | — | — | — | — | — | — | — | |
| TMDBackbone=RoBERTa-base2023.10 | 93.22 | 28.83 | 66.4 | 3,477.32 | 40.36 | 56 | 3,330.08 | 29.3 | 66.1 | 4,799.56 | — | — | — | -0.01 | — | — | — | — | |
| PGDtraining_type=adversarial training2023.06 | 93.2 | 26 | 72.1 | 1,562.8 | 21 | 77.6 | 2,114.6 | 41.6 | 53.2 | 905.8 | — | — | — | — | — | — | — | — | |
| FreeLBtraining_type=adversarial training2023.06 | 93.2 | 35 | 62.7 | 1,736.9 | 29 | 68.4 | 2,588.8 | 53 | 44.2 | 1,110.9 | — | — | — | — | — | — | — | — | |
| FreeLB++method_category=SOTA defense2023.06 | 93.2 | 45.3 | 51 | 1,895.3 | 39.9 | 56.9 | 2,732.5 | 42.9 | 54.6 | 1,094 | — | — | — | — | — | — | — | — | |
| PGD2024.10 | 93.2 | 26.2 | 69.2 | — | — | — | — | 17.4 | 81.6 | — | — | — | — | — | 32 | 65.8 | 27.2 | 69.6 | |
| AT2024.10 | 93.2 | 33.6 | 64.3 | — | — | — | — | 31.8 | 66.1 | — | — | — | — | — | 37.7 | 61.5 | 28.7 | 70.3 | |
| FREELB2024.10 | 93 | 25.1 | 73.6 | — | — | — | — | 19.9 | 76.9 | — | — | — | — | — | 40.9 | 55.5 | 42 | 54.7 | |
| TA-VAT2024.10 | 93 | 28.5 | 67.6 | — | — | — | — | 27.3 | 68.8 | — | — | — | — | — | 34.7 | 60.4 | 35.1 | 59.8 | |
| PRO-BERTnorm=HUBER2024.10 | 93 | 24.8 | 73.3 | — | — | — | — | 13.4 | 85.6 | — | — | — | — | — | 36.9 | 60.3 | 31.5 | 66.1 | |
| ALBERT2024.10 | 92.8 | 21.8 | 76.5 | — | — | — | — | 14.1 | 84.8 | — | — | — | — | — | 36.2 | 61 | 15.9 | 82.9 | |
| TAVAT2023.06 | 92.7 | 27.6 | 71.9 | 1,405.8 | 23.1 | 75.1 | 2,244.8 | 54.1 | 44.1 | 1,022.6 | — | — | — | — | — | — | — | — | |
| AdvFoolerBackbone=RoBERTa-base2023.10 | 92.69 | 32.02 | 63.7 | 3,255.35 | 47 | 49.4 | 3,141.88 | 33.55 | 62.2 | 4,205.77 | — | — | — | -0.54 | — | — | — | — | |
| ASCCBackbone=RoBERTa-base2023.10 | 92.62 | 77.92 | 20.2 | 606.14 | 83.57 | 15.2 | 548.34 | 84.31 | 14.7 | 770.35 | — | — | — | -0.61 | — | — | — | — | |
| RanMASKBackbone=BERT-base2023.10 | 92.61 | 61.13 | 35.8 | 2,392.46 | 64.33 | 33.1 | 2,463.24 | 66.38 | 31 | 2,740.39 | — | — | — | 0.47 | — | — | — | — | |
| FreeLBBackbone=BERT-base2023.10 | 92.33 | 68.35 | 28.9 | 670.2 | 76.12 | 21.8 | 549.85 | 71.74 | 25.8 | 776.31 | — | — | — | 0.19 | — | — | — | — | |
| BERT2024.10 | 92.3 | 11.8 | 87.2 | — | — | — | — | 11.3 | 87.4 | — | — | — | — | — | 32.8 | 64.5 | 26.4 | 71.5 | |
| BERT-baseBackbone=BERT-base2023.10 | 92.14 | 87.16 | 11.9 | 439.15 | 90.4 | 8.9 | 366.52 | 90.08 | 9.2 | 500.47 | — | — | — | — | — | — | — | — | |
| TMDBackbone=BERT-base2023.10 | 92.14 | 51.35 | 45.1 | 2,735.58 | 60.95 | 36.2 | 2,464.83 | 56.47 | 40.4 | 3,251.52 | — | — | — | 0 | — | — | — | — | |
| SAFERBackbone=BERT-base2023.10 | 92.12 | 45.64 | 50.5 | 1,262.35 | 52.74 | 43.1 | 1,133.61 | 50 | 46.1 | 1,455.25 | — | — | — | -0.02 | — | — | — | — | |
| Fine-tune2023.06 | 92.1 | 10.3 | 88.8 | 922.4 | 5.3 | 94.3 | 1,187 | 15.8 | 83.7 | 695.2 | — | — | — | — | — | — | — | — | |
| AdvFoolerBackbone=BERT-base2023.10 | 91.9 | 47.32 | 49.1 | 2,759.37 | 55.76 | 40.7 | 2,645.36 | 53.41 | 42.4 | 3,261.41 | — | — | — | -0.24 | — | — | — | — | |
| MIXADA2024.10 | 91.9 | 16.7 | 82 | — | — | — | — | 11.8 | 87.3 | — | — | — | — | — | 33.4 | 65.8 | 30 | 67.4 | |
| InfoBERTBackbone=BERT-base2023.10 | 91.71 | 72.47 | 25.3 | 645.03 | 77.26 | 20.9 | 510.46 | 75.52 | 22.5 | 719.98 | — | — | — | -0.43 | — | — | — | — | |
| DISTILBERT2024.10 | 90.9 | 14.9 | 83.6 | — | — | — | — | 4.3 | 95.3 | — | — | — | — | — | 18.8 | 79.3 | 9.6 | 89.4 | |
| ASCCBackbone=BERT-base2023.10 | 88.48 | 81.2 | 16.9 | 529.55 | 91.52 | 7.7 | 416.5 | 85.68 | 13 | 597.07 | — | — | — | -3.66 | — | — | — | — |