Harmful Question Answering on BeaverTails HarmfulQA (1k and 10k samples)
0Avg Harmfulness ScoreRandom
Evaluation results
| Method | Links | |
|---|---|---|
| Randomstatus=Pre-attack2024.05 | 0 | |
| Base: llama2-7b-chatstatus=Pre-attack2024.05 | 0.05 | |
| Security Vectorsstatus=Pre-attack2024.05 | 0.05 | |
| Vaccine (p = 1)status=Pre-attack2024.05 | 0.05 | |
| Vaccine (p = 10)status=Pre-attack2024.05 | 0.05 | |
| Additional safety trainingstatus=Pre-attack2024.05 | 0.05 | |
| Adversarial lossstatus=Pre-attack2024.05 | 0.05 | |
| RepNoisestatus=Pre-attack2024.05 | 0.05 | |
| Security Vectorsstatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.07 | |
| Security Vectorsstatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.08 | |
| RepNoisestatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.08 | |
| RepNoisestatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.1 | |
| RepNoisestatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.11 | |
| RepNoisestatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.12 | |
| RepNoisestatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.12 | |
| RepNoisestatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.13 | |
| Security Vectorsstatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.23 | |
| Gradient ascentstatus=Pre-attack2024.05 | 0.24 | |
| Adversarial lossstatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.26 | |
| Vaccine (p = 1)status=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.28 | |
| Vaccine (p = 10)status=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.28 | |
| Security Vectorsstatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.37 | |
| Gradient ascentstatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.38 | |
| Randomstatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.46 | |
| Base: llama2-7b-chatstatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.47 | |
| Randomstatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.47 | |
| Randomstatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.49 | |
| Security Vectorsstatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.52 | |
| Gradient ascentstatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.58 | |
| Adversarial lossstatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.64 | |
| Security Vectorsstatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.66 | |
| Gradient ascentstatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.68 | |
| Vaccine (p = 1)status=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.7 | |
| Adversarial lossstatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.7 | |
| Base: llama2-7b-chatstatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.72 | |
| Vaccine (p = 1)status=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.72 | |
| Vaccine (p = 10)status=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.72 | |
| Vaccine (p = 10)status=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.72 | |
| Base: llama2-7b-chatstatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.73 | |
| Base: llama2-7b-chatstatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.73 | |
| Vaccine (p = 1)status=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.73 | |
| Vaccine (p = 1)status=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.73 | |
| Vaccine (p = 10)status=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.73 | |
| Base: llama2-7b-chatstatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.74 | |
| Base: llama2-7b-chatstatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.74 | |
| Additional safety trainingstatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.74 | |
| Gradient ascentstatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.74 | |
| Gradient ascentstatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.74 | |
| Vaccine (p = 10)status=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.75 | |
| Additional safety trainingstatus=Post-attack, learning rate=3 x 10^-5, attack samples=1k2024.05 | 0.75 | |
| Additional safety trainingstatus=Post-attack, learning rate=6 x 10^-5, attack samples=1k2024.05 | 0.75 | |
| Additional safety trainingstatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.75 | |
| Adversarial lossstatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.75 | |
| Vaccine (p = 1)status=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.76 | |
| Vaccine (p = 10)status=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.76 | |
| Additional safety trainingstatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.76 | |
| Additional safety trainingstatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.76 | |
| Gradient ascentstatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.77 | |
| Adversarial lossstatus=Post-attack, learning rate=8 x 10^-5, attack samples=1k2024.05 | 0.77 | |
| Adversarial lossstatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.77 | |
| Randomstatus=Post-attack, learning rate=8 x 10^-5, attack samples=10k2024.05 | 0.82 | |
| Randomstatus=Post-attack, learning rate=6 x 10^-5, attack samples=10k2024.05 | 0.84 | |
| Randomstatus=Post-attack, learning rate=3 x 10^-5, attack samples=10k2024.05 | 0.86 |