Jailbreak Defense on GPTFuzz, TAP, GCG, AutoDAN, and Template
24.98GPTFuzz ASRSFT
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| SFTFine-tuned on=CodeAlpaca-20k2024.12 | 24.98 | 40 | 17.39 | 25.79 | 29.04 | 27.44 | 55.92 | |
| DAREFine-tuned on=CodeAlpaca-20k2024.12 | 26.16 | 42.38 | 19.02 | 25.5 | 28.91 | 28.4 | 54.56 | |
| SFTFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 32.11 | 44.76 | 23.91 | 36.96 | 38.52 | 35.25 | 47.71 | |
| DAREFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 32.51 | 41.43 | 23.91 | 36.1 | 38.58 | 34.51 | 48.05 | |
| SafeDecodingFine-tuned on=CodeAlpaca-20k2024.12 | 37.66 | 65.71 | 36.68 | 28.37 | 47.83 | 43.25 | 39.71 | |
| RESTAFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 54.51 | 76.19 | 89.13 | 86.82 | 78.24 | 76.98 | 5.98 | |
| RESTAdFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 55.1 | 75.71 | 88.86 | 84.24 | 77.49 | 76.28 | 6.68 | |
| IRRFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 59.56 | 82.86 | 85.05 | 91.12 | 79.68 | 79.65 | 3.31 | |
| IRRdFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 60.36 | 84.29 | 86.96 | 93.12 | 82.1 | 82.1 | 0.86 | |
| SFTfine-tuning dataset=Alpaca Chinese2024.12 | 66.2 | 46.19 | 34.78 | 64.47 | 69.42 | 56.21 | 26.75 | |
| DAREfine-tuning dataset=Alpaca Chinese2024.12 | 66.2 | 48.57 | 34.24 | 65.33 | 69.88 | 56.84 | 26.12 | |
| RESTAFine-tuned on=CodeAlpaca-20k2024.12 | 66.3 | 81.9 | 68.21 | 93.12 | 83.47 | 78.6 | 4.36 | |
| RESTAdFine-tuned on=CodeAlpaca-20k2024.12 | 66.8 | 82.86 | 69.57 | 91.69 | 83.63 | 78.91 | 4.05 | |
| IRRFine-tuned on=CodeAlpaca-20k2024.12 | 71.46 | 85.24 | 67.93 | 91.69 | 89.28 | 81.12 | 1.84 | |
| Safe LoRAfine-tuning dataset=Alpaca Chinese2024.12 | 74.73 | 96.19 | 89.13 | 65.04 | 89.58 | 82.93 | 0.03 | |
| Safe LoRAFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 74.73 | 96.19 | 89.13 | 65.04 | 89.58 | 82.93 | 0.03 | |
| IRRdFine-tuned on=CodeAlpaca-20k2024.12 | 75.12 | 87.62 | 71.2 | 93.12 | 90.59 | 83.53 | -0.57 | |
| Safe LORAFine-tuned on=CodeAlpaca-20k2024.12 | 77.6 | 96.67 | 60.87 | 80.8 | 80.82 | 79.35 | 3.61 | |
| SafeDecodingFine-tuned on=GSM8K, Safety Realignment=true2024.12 | 78.39 | 85.24 | 92.12 | 52.44 | 52.44 | 76.35 | 6.05 | |
| SafeDecodingfine-tuning dataset=Alpaca Chinese2024.12 | 81.27 | 66.19 | 62.77 | 76.22 | 77.79 | 72.85 | 10.11 | |
| RESTAdfine-tuning dataset=Alpaca Chinese2024.12 | 89.89 | 77.62 | 79.35 | 99.14 | 95.95 | 88.39 | -5.43 | |
| RESTAfine-tuning dataset=Alpaca Chinese2024.12 | 91.28 | 78.57 | 79.89 | 98.85 | 96.01 | 88.92 | -5.96 | |
| IRRfine-tuning dataset=Alpaca Chinese2024.12 | 91.58 | 78.1 | 62.23 | 99.43 | 95.23 | 85.31 | -2.35 | |
| IRRdfine-tuning dataset=Alpaca Chinese2024.12 | 92.17 | 81.9 | 62.23 | 100 | 95.95 | 86.45 | -3.49 |