Defense against Indirect Prompt Injection on Filtered QA dataset
97.65ASR (Naive)Reminder
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ReminderVictim Model=Qwen2-7b-Instruct2024.11 | 97.65 | 97.95 | 100 | 100 | 100 | |
| InstructionalVictim Model=Qwen2-7b-Instruct2024.11 | 95.55 | 95.75 | 99.95 | 100 | 100 | |
| SpotlightVictim Model=Qwen2-7b-Instruct2024.11 | 94.35 | 96.45 | 100 | 100 | 100 | |
| SpotlightVictim Model=Llama3.1-8b-Instruct2024.11 | 94.35 | 96.45 | 100 | 100 | 100 | |
| NoneVictim Model=Qwen2-7b-Instruct2024.11 | 92.45 | 95.9 | 100 | 100 | 100 | |
| NoneVictim Model=Llama3.1-8b-Instruct2024.11 | 85.9 | 91.1 | 81.7 | 95.25 | 92.3 | |
| ReminderVictim Model=Llama3.1-8b-Instruct2024.11 | 79.05 | 77.3 | 71.75 | 84.35 | 80.65 | |
| IsolationVictim Model=Qwen2-7b-Instruct2024.11 | 77.8 | 88.85 | 99.35 | 99.7 | 100 | |
| IsolationVictim Model=Llama3.1-8b-Instruct2024.11 | 76.75 | 85 | 89.75 | 91.7 | 88.75 | |
| InstructionalVictim Model=Llama3.1-8b-Instruct2024.11 | 60.15 | 68.35 | 88.1 | 84.7 | 84.85 | |
| NoneVictim Model=Llama3-8b-Instruct2024.11 | 10.55 | 53.35 | 88.25 | 75.3 | 86 | |
| ReminderVictim Model=Llama3-8b-Instruct2024.11 | 10.55 | 39.9 | 67.5 | 37.85 | 51.2 | |
| SpotlightVictim Model=Llama3-8b-Instruct2024.11 | 8.8 | 32.85 | 76.35 | 74.45 | 56.6 | |
| InstructionalVictim Model=Llama3-8b-Instruct2024.11 | 6.95 | 35 | 80.1 | 64.45 | 62.75 | |
| SandwichVictim Model=Qwen2-7b-Instruct2024.11 | 4.8 | 6.15 | 14 | 34.2 | 34.6 | |
| SandwichVictim Model=Llama3.1-8b-Instruct2024.11 | 2.5 | 3.05 | 22.9 | 3.35 | 9.55 | |
| IsolationVictim Model=Llama3-8b-Instruct2024.11 | 2.2 | 33.75 | 83.35 | 67.4 | 77.75 | |
| Fake Completion DefenseVictim Model=Llama3.1-8b-Instruct2024.11 | 1.75 | 2.45 | 8.75 | 0.8 | 0.6 | |
| Escape DefenseVictim Model=Qwen2-7b-Instruct2024.11 | 1.45 | 1.7 | 0.75 | 0.68 | 4.95 | |
| Escape DefenseVictim Model=Llama3.1-8b-Instruct2024.11 | 1.25 | 2.7 | 1.05 | 0.9 | 1.65 | |
| Ignore DefenseVictim Model=Qwen2-7b-Instruct2024.11 | 0.85 | 0.7 | 0.8 | 0.95 | 4.1 | |
| SandwichVictim Model=Llama3-8b-Instruct2024.11 | 0.45 | 9.35 | 49.55 | 7.3 | 21.25 | |
| Fake Completion DefenseVictim Model=Qwen2-7b-Instruct2024.11 | 0.3 | 0.7 | 0.55 | 0.45 | 0.3 | |
| Escape DefenseVictim Model=Llama3-8b-Instruct2024.11 | 0.25 | 1.7 | 1.05 | 0.55 | 1.45 | |
| Ignore DefenseVictim Model=Llama3.1-8b-Instruct2024.11 | 0.25 | 0.3 | 0.35 | 0.45 | 1.1 | |
| Fake Completion DefenseVictim Model=Llama3-8b-Instruct2024.11 | 0.1 | 1.8 | 17.7 | 0.05 | 0.1 | |
| Fake Completion Defense with TemplateVictim Model=Llama3.1-8b-Instruct2024.11 | 0.1 | 0.25 | 0.2 | 0.15 | 0.05 | |
| Ignore DefenseVictim Model=Llama3-8b-Instruct2024.11 | 0.05 | 0.35 | 0.3 | 0.1 | 1.35 | |
| Fake Completion Defense with TemplateVictim Model=Llama3-8b-Instruct2024.11 | 0.05 | 0.05 | 0.3 | 0.05 | 0.05 | |
| Fake Completion Defense with TemplateVictim Model=Qwen2-7b-Instruct2024.11 | 0.05 | 0.25 | 0.2 | 0.15 | 0.05 |