Prompt Injection Defense on Inj-SQuAD
0.11Combined ASRStruQ
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| StruQModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 0.11 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 0.11 | — | — | — | — | |
| Robustness via ReferencingBase Model=GPT-4o-mini2025.04 | 0.22 | 0.22 | 0.22 | 0.22 | 0.22 | |
| Robustness via ReferencingModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 0.22 | — | — | — | — | |
| StruQModel=Llama3.1-8B-Instruct, Attack Method=Escape2025.04 | 0.22 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3.1-8B-Instruct, Attack Method=Escape2025.04 | 0.22 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3.1-8B-Instruct, Attack Method=Fakecom2025.04 | 0.22 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3.1-8B-Instruct, Attack Method=Combined2025.04 | 0.22 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 0.33 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 0.56 | — | — | — | — | |
| Robustness via ReferencingBase Model=GPT-4o2025.04 | 0.67 | 1 | 1.22 | 1 | 0.44 | |
| Robustness via ReferencingModel=Llama3-8B-Instruct, Attack Method=Combined2025.04 | 0.78 | — | — | — | — | |
| StruQModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 1.11 | — | — | — | — | |
| Robustness via ReferencingBase Model=GPT-4.12025.04 | 1.22 | 0.67 | 1.89 | 0.78 | 0.44 | |
| Robustness via ReferencingModel=Llama3-8B-Instruct, Attack Method=Fakecom2025.04 | 1.22 | — | — | — | — | |
| Robustness via ReferencingModel=Qwen2-7B-Instruct, Attack Method=Combined2025.04 | 1.44 | — | — | — | — | |
| Robustness via ReferencingModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 1.56 | — | — | — | — | |
| Robustness via ReferencingModel=Qwen2-7B-Instruct, Attack Method=Fakecom2025.04 | 1.78 | — | — | — | — | |
| Robustness via ReferencingModel=Qwen2-7B-Instruct, Attack Method=Ignore2025.04 | 2.33 | — | — | — | — | |
| Robustness via ReferencingModel=Qwen2-7B-Instruct, Attack Method=Escape2025.04 | 2.56 | — | — | — | — | |
| StruQModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 3.33 | — | — | — | — | |
| StruQModel=Llama3-8B-Instruct, Attack Method=Fakecom2025.04 | 3.33 | — | — | — | — | |
| StruQModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 4 | — | — | — | — | |
| Robustness via ReferencingModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 4 | — | — | — | — | |
| StruQModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 4.22 | — | — | — | — | |
| SandwichBase Model=GPT-4o2025.04 | 9 | 8.89 | 2.56 | 8.67 | 9.78 | |
| StruQModel=Qwen2-7B-Instruct, Attack Method=Escape2025.04 | 11.11 | — | — | — | — | |
| StruQModel=Qwen2-7B-Instruct, Attack Method=Ignore2025.04 | 11.22 | — | — | — | — | |
| StruQModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 12.78 | — | — | — | — | |
| SandwichBase Model=GPT-4.12025.04 | 16.22 | 9.89 | 5.56 | 9.22 | 9.78 | |
| StruQModel=Llama3-8B-Instruct, Attack Method=Combined2025.04 | 16.67 | — | — | — | — | |
| SpotlightBase Model=GPT-4.12025.04 | 17.33 | 10.11 | 2.89 | 12.67 | 26.78 | |
| SandwichBase Model=GPT-4o-mini2025.04 | 19 | 19.89 | 8.56 | 18.89 | 14.56 | |
| SandwichModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 19.67 | — | — | — | — | |
| InstructionalBase Model=GPT-4o-mini2025.04 | 21.56 | 27.89 | 6.67 | 34 | 34 | |
| SpotlightBase Model=GPT-4o2025.04 | 22.67 | 11.44 | 5.11 | 13.56 | 28.33 | |
| SandwichModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 23.67 | — | — | — | — | |
| SandwichModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 23.89 | — | — | — | — | |
| SandwichModel=Llama3-8B-Instruct, Attack Method=Fakecom2025.04 | 25.89 | — | — | — | — | |
| SandwichModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 27.67 | — | — | — | — | |
| InstructionalBase Model=GPT-4o2025.04 | 30.11 | 8.33 | 1.33 | 12.22 | 43.33 | |
| SandwichModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 30.56 | — | — | — | — | |
| SandwichModel=Llama3.1-8B-Instruct, Attack Method=Fakecom2025.04 | 30.89 | — | — | — | — | |
| SpotlightModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 31 | — | — | — | — | |
| SandwichModel=Qwen2-7B-Instruct, Attack Method=Ignore2025.04 | 33.11 | — | — | — | — | |
| SpotlightModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 33.11 | — | — | — | — | |
| SandwichModel=Qwen2-7B-Instruct, Attack Method=Escape2025.04 | 34.11 | — | — | — | — | |
| ReminderBase Model=GPT-4o2025.04 | 34.78 | 11 | 1.11 | 14 | 34.78 | |
| InstructionalBase Model=GPT-4.12025.04 | 36.67 | 11.67 | 2.89 | 15.22 | 21.22 | |
| SandwichModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 38.11 | — | — | — | — | |
| SandwichModel=Llama3.1-8B-Instruct, Attack Method=Escape2025.04 | 39.11 | — | — | — | — | |
| ReminderBase Model=GPT-4.12025.04 | 41.44 | 16.89 | 1.67 | 17.22 | 7.44 | |
| SandwichModel=Llama3.1-8B-Instruct, Attack Method=Combined2025.04 | 42.22 | — | — | — | — | |
| ReminderBase Model=GPT-4o-mini2025.04 | 43.44 | 29.44 | 17.33 | 42.89 | 33.33 | |
| StruQModel=Llama3.1-8B-Instruct, Attack Method=Fakecom2025.04 | 46.22 | — | — | — | — | |
| SpotlightModel=Llama3.1-8B-Instruct, Attack Method=Escape2025.04 | 46.89 | — | — | — | — | |
| InstructionalModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 47.78 | — | — | — | — | |
| InstructionalModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 48.78 | — | — | — | — | |
| SpotlightModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 49.11 | — | — | — | — | |
| SandwichModel=Llama3-8B-Instruct, Attack Method=Combined2025.04 | 49.89 | — | — | — | — | |
| SandwichModel=Qwen2-7B-Instruct, Attack Method=Combined2025.04 | 52 | — | — | — | — | |
| InstructionalModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 52.33 | — | — | — | — | |
| ReminderModel=Llama3-8B-Instruct, Attack Method=Fakecom2025.04 | 52.67 | — | — | — | — | |
| SpotlightModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 52.67 | — | — | — | — | |
| SandwichModel=Qwen2-7B-Instruct, Attack Method=Fakecom2025.04 | 52.67 | — | — | — | — | |
| NoneModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 53.56 | — | — | — | — | |
| SpotlightModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 54 | — | — | — | — | |
| StruQModel=Llama3.1-8B-Instruct, Attack Method=Combined2025.04 | 56 | — | — | — | — | |
| ReminderModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 58.89 | — | — | — | — | |
| SpotlightModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 60.78 | — | — | — | — | |
| InstructionalModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 61.89 | — | — | — | — | |
| SpotlightModel=Qwen2-7B-Instruct, Attack Method=Ignore2025.04 | 63.67 | — | — | — | — | |
| InstructionalModel=Llama3-8B-Instruct, Attack Method=Combined2025.04 | 63.78 | — | — | — | — | |
| ReminderModel=Llama3-8B-Instruct, Attack Method=Naive2025.04 | 64.11 | — | — | — | — | |
| ReminderModel=Llama3-8B-Instruct, Attack Method=Combined2025.04 | 64.44 | — | — | — | — | |
| NoneModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 64.44 | — | — | — | — | |
| InstructionalModel=Llama3-8B-Instruct, Attack Method=Fakecom2025.04 | 66.89 | — | — | — | — | |
| SpotlightModel=Qwen2-7B-Instruct, Attack Method=Escape2025.04 | 67.44 | — | — | — | — | |
| InstructionalModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 70.11 | — | — | — | — | |
| InstructionalModel=Llama3.1-8B-Instruct, Attack Method=Escape2025.04 | 70.44 | — | — | — | — | |
| NoneModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 70.67 | — | — | — | — | |
| InstructionalModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 71.11 | — | — | — | — | |
| SpotlightBase Model=GPT-4o-mini2025.04 | 71.78 | 18.11 | 21.56 | 17.22 | 75.67 | |
| NoneModel=Llama3-8B-Instruct, Attack Method=Ignore2025.04 | 73.22 | — | — | — | — | |
| ReminderModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 73.67 | — | — | — | — | |
| NoneModel=Llama3-8B-Instruct, Attack Method=Escape2025.04 | 75.11 | — | — | — | — | |
| NoneModel=Llama3.1-8B-Instruct, Attack Method=Escape2025.04 | 76.67 | — | — | — | — | |
| InstructionalModel=Qwen2-7B-Instruct, Attack Method=Ignore2025.04 | 77 | — | — | — | — | |
| NoneModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 77.56 | — | — | — | — | |
| ReminderModel=Llama3.1-8B-Instruct, Attack Method=Ignore2025.04 | 77.56 | — | — | — | — | |
| InstructionalModel=Llama3.1-8B-Instruct, Attack Method=Combined2025.04 | 77.56 | — | — | — | — | |
| SpotlightModel=Llama3-8B-Instruct, Attack Method=Combined2025.04 | 78.56 | — | — | — | — | |
| StruQModel=Qwen2-7B-Instruct, Attack Method=Fakecom2025.04 | 78.56 | — | — | — | — | |
| InstructionalModel=Qwen2-7B-Instruct, Attack Method=Escape2025.04 | 78.78 | — | — | — | — | |
| NoneModel=Qwen2-7B-Instruct, Attack Method=Escape2025.04 | 78.89 | — | — | — | — | |
| ReminderModel=Qwen2-7B-Instruct, Attack Method=Naive2025.04 | 79.22 | — | — | — | — | |
| InstructionalModel=Llama3.1-8B-Instruct, Attack Method=Fakecom2025.04 | 79.56 | — | — | — | — | |
| NoneModel=Qwen2-7B-Instruct, Attack Method=Ignore2025.04 | 80.11 | — | — | — | — | |
| ReminderModel=Llama3.1-8B-Instruct, Attack Method=Naive2025.04 | 80.67 | — | — | — | — | |
| StruQModel=Qwen2-7B-Instruct, Attack Method=Combined2025.04 | 82.78 | — | — | — | — |