Direct Prompt Injection on AlpacaFarm (208 samples)
78.36Naive Success RateReminder
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ReminderVictim Model=Qwen2-7b-Instruct2024.11 | 78.36 | 87.01 | 90.38 | 99.51 | 100 | |
| NoneVictim Model=Qwen2-7b-Instruct2024.11 | 74.03 | 85.09 | 90.86 | 100 | 100 | |
| InstructionalVictim Model=Qwen2-7b-Instruct2024.11 | 74.03 | 85.09 | 83.17 | 99.03 | 100 | |
| SpotlightVictim Model=Qwen2-7b-Instruct2024.11 | 74.03 | 78.84 | 77.4 | 99.51 | 99.51 | |
| IsolationVictim Model=Qwen2-7b-Instruct2024.11 | 58.17 | 73.55 | 79.8 | 96.15 | 98.55 | |
| NoneVictim Model=Llama3.1-8b-Instruct2024.11 | 51.92 | 76.92 | 62.98 | 79.8 | 77.4 | |
| IsolationVictim Model=Llama3.1-8b-Instruct2024.11 | 46.63 | 67.3 | 59.13 | 77.88 | 64.42 | |
| NoneVictim Model=Llama3-8b-Instruct2024.11 | 46.15 | 74.51 | 54.8 | 64.9 | 76.92 | |
| InstructionalVictim Model=Llama3.1-8b-Instruct2024.11 | 39.42 | 48.55 | 51.44 | 62.01 | 47.11 | |
| SpotlightVictim Model=Llama3.1-8b-Instruct2024.11 | 38.94 | 57.69 | 41.34 | 68.75 | 68.75 | |
| IsolationVictim Model=Llama3-8b-Instruct2024.11 | 37.98 | 64.9 | 47.11 | 62.01 | 75.48 | |
| InstructionalVictim Model=Llama3-8b-Instruct2024.11 | 36.53 | 35.57 | 48.07 | 31.25 | 29.32 | |
| ReminderVictim Model=Llama3.1-8b-Instruct2024.11 | 35.57 | 56.25 | 39.42 | 36.53 | 42.3 | |
| SpotlightVictim Model=Llama3-8b-Instruct2024.11 | 27.88 | 53.36 | 45.19 | 75.96 | 66.34 | |
| SandwichVictim Model=Qwen2-7b-Instruct2024.11 | 27.4 | 47.11 | 29.8 | 52.4 | 67.78 | |
| Fake Completion DefenseVictim Model=Llama3.1-8b-Instruct2024.11 | 27.4 | 33.17 | 22.11 | 7.21 | 17.3 | |
| ReminderVictim Model=Llama3-8b-Instruct2024.11 | 24.51 | 37.5 | 36.05 | 16.82 | 35.09 | |
| SandwichVictim Model=Llama3.1-8b-Instruct2024.11 | 22.59 | 32.69 | 22.59 | 33.17 | 34.13 | |
| SandwichVictim Model=Llama3-8b-Instruct2024.11 | 21.63 | 38.46 | 20.67 | 18.75 | 49.51 | |
| Escape DefenseVictim Model=Qwen2-7b-Instruct2024.11 | 21.63 | 29.32 | 16.82 | 70.19 | 36.53 | |
| Fake Completion DefenseVictim Model=Qwen2-7b-Instruct2024.11 | 20.67 | 13.94 | 13.46 | 3.36 | 6.25 | |
| Escape DefenseVictim Model=Llama3-8b-Instruct2024.11 | 19.71 | 38.94 | 14.9 | 25 | 34.61 | |
| Fake Completion DefenseVictim Model=Llama3-8b-Instruct2024.11 | 16.82 | 36.53 | 12.5 | 0.48 | 6.25 | |
| Ignore DefenseVictim Model=Llama3.1-8b-Instruct2024.11 | 12.5 | 13.94 | 5.76 | 8.17 | 9.13 | |
| Escape DefenseVictim Model=Llama3.1-8b-Instruct2024.11 | 12.5 | 13.94 | 5.76 | 8.17 | 9.13 | |
| Ignore DefenseVictim Model=Qwen2-7b-Instruct2024.11 | 12.01 | 11.53 | 8.65 | 5.28 | 16.34 | |
| Fake Completion Defense with TemplateVictim Model=Llama3-8b-Instruct2024.11 | 11.53 | 5.28 | 7.21 | 0 | 1.44 | |
| Ignore DefenseVictim Model=Llama3-8b-Instruct2024.11 | 11.05 | 22.11 | 7.21 | 7.69 | 27.4 | |
| Fake Completion Defense with TemplateVictim Model=Qwen2-7b-Instruct2024.11 | 11.05 | 7.21 | 8.17 | 4.32 | 2.4 | |
| Fake Completion Defense with TemplateVictim Model=Llama3.1-8b-Instruct2024.11 | 9.13 | 4.32 | 3.36 | 2.4 | 3.84 |