Indirect Prompt Injection Attack on IPI-3k
69.3ASRAdapTools
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AdapToolsFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=No Defense2026.02 | 69.3 | 6.7 | |
| AdapToolsFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=No Defense2026.02 | 60.6 | 32.6 | |
| AutoHijackerFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=No Defense2026.02 | 47.4 | 12.4 | |
| AdapToolsFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=No Defense2026.02 | 44.4 | 15.2 | |
| AdapToolsFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=Pi-Detector2026.02 | 41.3 | 24.3 | |
| Ignore InstructionFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=No Defense2026.02 | 39.4 | 12.8 | |
| AdapToolsFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=Pi-Detector2026.02 | 39.1 | 50 | |
| AutoHijackerFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=No Defense2026.02 | 37.6 | 17.8 | |
| Combined AttackFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=No Defense2026.02 | 37.4 | 14.2 | |
| AdapToolsFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=MELON2026.02 | 36.4 | 27.3 | |
| AdapToolsFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=No Defense2026.02 | 35.4 | 17.3 | |
| InjectAgentFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=No Defense2026.02 | 35.2 | 14.4 | |
| InjectAgentFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=No Defense2026.02 | 34.8 | 16 | |
| AdapToolsFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=MELON2026.02 | 33.7 | 54.3 | |
| Combined AttackFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=No Defense2026.02 | 32.8 | 17.8 | |
| AutoHijackerFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=No Defense2026.02 | 29.6 | 49.6 | |
| AutoHijackerFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=Pi-Detector2026.02 | 27.6 | 19.5 | |
| AdapToolsFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=Pi-Detector2026.02 | 26.7 | 19.4 | |
| AdapToolsFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=No Defense2026.02 | 26.1 | 44.8 | |
| Ignore InstructionFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=No Defense2026.02 | 25.2 | 23 | |
| AutoHijackerFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=Pi-Detector2026.02 | 23.8 | 20.3 | |
| AdapToolsFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=Pi-Detector2026.02 | 23.7 | 24.4 | |
| Ignore InstructionFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=Pi-Detector2026.02 | 23.2 | 18.6 | |
| InjectAgentFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=Pi-Detector2026.02 | 23 | 19 | |
| AdapToolsFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=MELON2026.02 | 23 | 20.9 | |
| Combined AttackFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=Pi-Detector2026.02 | 22.6 | 19.5 | |
| AutoHijackerFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=No Defense2026.02 | 22.4 | 26 | |
| InjectAgentFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=Pi-Detector2026.02 | 20.6 | 21.1 | |
| Combined AttackFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=Pi-Detector2026.02 | 20.4 | 22.5 | |
| AdapToolsFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=No Defense2026.02 | 20.3 | 36.8 | |
| Combined AttackFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=No Defense2026.02 | 19.4 | 56.4 | |
| AdapToolsFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=MELON2026.02 | 18.5 | 27.6 | |
| Combined AttackFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=No Defense2026.02 | 18.2 | 28.2 | |
| AutoHijackerFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=Pi-Detector2026.02 | 16.4 | 60.2 | |
| Ignore InstructionFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=Pi-Detector2026.02 | 16.2 | 26.4 | |
| AdapToolsFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=Pi-Detector2026.02 | 16.1 | 51.1 | |
| InjectAgentFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=No Defense2026.02 | 15 | 60.6 | |
| AutoHijackerFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=MELON2026.02 | 14.8 | 24.1 | |
| Ignore InstructionFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=No Defense2026.02 | 13.6 | 61 | |
| AdapToolsFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=Pi-Detector2026.02 | 13.5 | 40.2 | |
| AutoHijackerFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=Pi-Detector2026.02 | 13.4 | 31.5 | |
| AdapToolsFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=MELON2026.02 | 13.3 | 52.9 | |
| AutoHijackerFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=MELON2026.02 | 13.2 | 24.6 | |
| AutoHijackerFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=No Defense2026.02 | 12 | 55.4 | |
| Ignore InstructionFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=MELON2026.02 | 11.4 | 22.9 | |
| Combined AttackFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=MELON2026.02 | 11.4 | 23.6 | |
| Combined AttackFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=Pi-Detector2026.02 | 11 | 63.1 | |
| InjectAgentFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=MELON2026.02 | 10.8 | 23.7 | |
| Combined AttackFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=Pi-Detector2026.02 | 10 | 33.2 | |
| InjectAgentFM=MISTRAL-8B, CoT=None-reasoning, BU=36, Defense=MELON2026.02 | 9.6 | 25.1 | |
| Combined AttackFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=MELON2026.02 | 9.4 | 26.7 | |
| AutoHijackerFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=No Defense2026.02 | 9.3 | 40.2 | |
| Ignore InstructionFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=No Defense2026.02 | 9.2 | 31.4 | |
| InjectAgentFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=No Defense2026.02 | 9.2 | 33.2 | |
| InjectAgentFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=Pi-Detector2026.02 | 8.2 | 66 | |
| Combined AttackFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=No Defense2026.02 | 8 | 54.4 | |
| Ignore InstructionFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=Pi-Detector2026.02 | 8 | 65.5 | |
| Ignore InstructionFM=LLAMA-3.1-8B, CoT=None-reasoning, BU=38, Defense=MELON2026.02 | 7.8 | 29.6 | |
| AutoHijackerFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=MELON2026.02 | 7.4 | 67.4 | |
| AutoHijackerFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=Pi-Detector2026.02 | 6.8 | 59 | |
| AdapToolsFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=MELON2026.02 | 6.7 | 43.6 | |
| AutoHijackerFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=Pi-Detector2026.02 | 6.5 | 40.8 | |
| AutoHijackerFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=MELON2026.02 | 6.2 | 35.9 | |
| Ignore InstructionFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=Pi-Detector2026.02 | 5.2 | 33.8 | |
| Combined AttackFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=MELON2026.02 | 5 | 36.3 | |
| InjectAgentFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=Pi-Detector2026.02 | 4.8 | 35.9 | |
| Combined AttackFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=MELON2026.02 | 4.8 | 68.1 | |
| Combined AttackFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=Pi-Detector2026.02 | 4.6 | 56.6 | |
| Ignore InstructionFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=MELON2026.02 | 4.4 | 68.4 | |
| InjectAgentFM=QWEN-3-8B, CoT=Reasoning, BU=80, Defense=MELON2026.02 | 4.4 | 69.1 | |
| AutoHijackerFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=MELON2026.02 | 4.3 | 41.3 | |
| InjectAgentFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=MELON2026.02 | 3.4 | 36.7 | |
| Ignore InstructionFM=GEMINI-2.5-FLASH, CoT=None-reasoning, BU=61, Defense=MELON2026.02 | 3.2 | 35.1 | |
| AutoHijackerFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=MELON2026.02 | 3 | 61.5 | |
| Combined AttackFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=MELON2026.02 | 2.4 | 58.1 | |
| InjectAgentFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=No Defense2026.02 | 1.8 | 60.4 | |
| Ignore InstructionFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=No Defense2026.02 | 1.4 | 55.4 | |
| Ignore InstructionFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=Pi-Detector2026.02 | 0.8 | 55.8 | |
| InjectAgentFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=Pi-Detector2026.02 | 0.8 | 61.1 | |
| Ignore InstructionFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=MELON2026.02 | 0.4 | 56.1 | |
| InjectAgentFM=GPT-4.1, CoT=None-reasoning, BU=66, Defense=MELON2026.02 | 0.4 | 61.3 | |
| Ignore InstructionFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=No Defense2026.02 | 0.4 | 39.8 | |
| Combined AttackFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=No Defense2026.02 | 0.4 | 37.2 | |
| InjectAgentFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=No Defense2026.02 | 0.4 | 38 | |
| Ignore InstructionFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=Pi-Detector2026.02 | 0.2 | 39.9 | |
| Combined AttackFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=MELON2026.02 | 0.2 | 37.3 | |
| Combined AttackFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=Pi-Detector2026.02 | 0.2 | 37.3 | |
| InjectAgentFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=MELON2026.02 | 0.2 | 39.1 | |
| InjectAgentFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=Pi-Detector2026.02 | 0.2 | 38.1 | |
| Ignore InstructionFM=DEEPSEEK-R1, CoT=Reasoning, BU=50, Defense=MELON2026.02 | 0 | 40 |