Safeguarding LLM Agents against Prompt Injection on Banking and Slack (test)
91.9BU (No Attack)Repeated-Prompt
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Repeated-PromptLLM backbone=Qwen32026.05 | 91.9 | 0.7 | 8.9 | 0.4 | 5.4 | 0.3 | 89.3 | 0.4 | 80.4 | 0.6 | 92.9 | 2 | 55.4 | 0.8 | |
| SpotlightingLLM backbone=Qwen32026.05 | 89.2 | 0.4 | 5.4 | 0.2 | 12.5 | 0.4 | 94.6 | 0.3 | 91.1 | 0.1 | 96.4 | 0.6 | 60 | 0.3 | |
| No DefenseLLM backbone=Qwen32026.05 | 86.5 | 0 | 10.7 | 0 | 8.9 | 0 | 87.5 | 0 | 94.6 | 0 | 92.9 | 0 | 58.9 | 0 | |
| MAGELLM backbone=Qwen32026.05 | 86.5 | 4.8 | 0 | 4.1 | 0 | 3.4 | 0 | 3.4 | 0 | 3.9 | 0 | 2.8 | 0 | 3.5 | |
| Repeated-PromptLLM backbone=Llama-3.32026.05 | 83.8 | 7.2 | 10.7 | 16.3 | 21.4 | 17 | 17.9 | 23.6 | 32.1 | 13.7 | 76.8 | 14.3 | 31.8 | 17 | |
| IPIGuardLLM backbone=Qwen32026.05 | 78.4 | 3.8 | 0 | 2.1 | 0 | 2.1 | 0 | 1.1 | 1.8 | 4.5 | 1.8 | 9.1 | 0.7 | 3.79 | |
| No DefenseLLM backbone=Llama-3.32026.05 | 75.7 | 0 | 26.8 | 0 | 35.7 | 0 | 44.6 | 0 | 71.4 | 0 | 92.9 | 0 | 54.3 | 0 | |
| SpotlightingLLM backbone=Llama-3.32026.05 | 75.7 | 0.2 | 30.4 | 0 | 25 | 0 | 44.6 | 0.4 | 60.7 | 0 | 85.7 | 0 | 49.3 | 0.1 | |
| MELONLLM backbone=Llama-3.32026.05 | 75.7 | 3.1 | 23.2 | 3.2 | 19.6 | 3.1 | 32.1 | 4.3 | 57.1 | 4.6 | 78.6 | 8.8 | 42.1 | 4.8 | |
| MELONLLM backbone=Qwen32026.05 | 75.7 | 3.4 | 3.6 | 2.5 | 3.6 | 2.7 | 0 | 1.7 | 37.5 | 2.8 | 58.9 | 6.8 | 20.7 | 3.3 | |
| IPIGuardLLM backbone=Llama-3.32026.05 | 73 | 3 | 0 | 2.9 | 0 | 2.8 | 0 | 4.5 | 1.8 | 11.8 | 3.6 | 4.3 | 1.1 | 5.3 | |
| MAGELLM backbone=Llama-3.32026.05 | 73 | 4.8 | 0 | 3.1 | 0 | 3 | 1.8 | 3.5 | 0 | 2.7 | 0 | 2.9 | 0.4 | 3 | |
| Repeated-PromptLLM backbone=GPT-4o-mini2026.05 | 64.9 | 7.8 | 7.1 | 4.6 | 7.1 | 9.4 | 25 | 7.1 | 16.1 | 5.2 | 71.4 | 15.4 | 25.4 | 8.3 | |
| No DefenseLLM backbone=GPT-4o-mini2026.05 | 62.2 | 0 | 14.3 | 0 | 21.4 | 0 | 42.9 | 0 | 69.6 | 0 | 98.2 | 0 | 49.3 | 0 | |
| MAGELLM backbone=GPT-4o-mini2026.05 | 59.5 | 3.4 | 0 | 2.8 | 0 | 3 | 3.4 | 3.1 | 0 | 2.9 | 0 | 2.7 | 0.7 | 2.9 | |
| SpotlightingLLM backbone=GPT-4o-mini2026.05 | 59.4 | 0 | 14.3 | 0 | 17.9 | 0 | 39.3 | 0 | 60.7 | 0 | 98.2 | 8.7 | 46.1 | 1.7 | |
| MELONLLM backbone=GPT-4o-mini2026.05 | 56.8 | 3.9 | 0 | 3.2 | 0 | 3.5 | 0 | 4 | 3.6 | 2.8 | 0 | 2.3 | 0.7 | 3.2 | |
| IPIGuardLLM backbone=GPT-4o-mini2026.05 | 56.8 | 4.2 | 0 | 4.7 | 0 | 4.6 | 0 | 5.7 | 3.6 | 7.7 | 0 | 11 | 0.7 | 6.8 | |
| PI-DetectorLLM backbone=Llama-3.32026.05 | 29.7 | 0.3 | 0 | 0.7 | 0 | 0.7 | 17.9 | 1.2 | 17.9 | 1 | 16.1 | 1.1 | 10.4 | 0.9 | |
| PI-DetectorLLM backbone=GPT-4o-mini2026.05 | 24.3 | 0.3 | 0 | 0.7 | 0 | 0.6 | 8.9 | 1.3 | 12.5 | 1 | 35.7 | 1.5 | 11.4 | 1 | |
| PI-DetectorLLM backbone=Qwen32026.05 | 24.3 | 0.2 | 0 | 0.3 | 0 | 0.3 | 14.3 | 0.6 | 12.5 | 0.5 | 3.6 | 0.6 | 6.1 | 0.5 |