Agent Task Performance on AgentDojo Banking
62.5Attack Success RateNo Defense
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| No DefenseModel=GPT-4o, Attack Type=Important Instructions2024.12 | 62.5 | 69.44 | — | — | |
| No DefenseModel=GPT-4o2024.12 | 62.5 | — | 75 | 69.44 | |
| DelimitingModel=GPT-4o2024.12 | 61.81 | — | 81.25 | 77.08 | |
| Repeat PromptModel=GPT-4o2024.12 | 46.53 | — | 93.75 | 77.08 | |
| No DefenseModel=GPT-4o-mini2024.12 | 34.03 | — | 50 | 38.19 | |
| DelimitingModel=GPT-4o-mini2024.12 | 33.33 | — | 43.75 | 35.42 | |
| Repeat PromptModel=GPT-4o-mini2024.12 | 17.36 | — | 43.75 | 43.75 | |
| No DefenseModel=GPT-4o, Attack Type=Injecagent2024.12 | 15.28 | 72.22 | — | — | |
| Tool FilterModel=GPT-4o2024.12 | 11.11 | — | 81.25 | 56.94 | |
| Tool FilterModel=GPT-4o-mini2024.12 | 11.11 | — | 50 | 43.06 | |
| PI DetectorModel=GPT-4o-mini2024.12 | 10.42 | — | 37.5 | 29.86 | |
| No DefenseModel=GPT-4o, Attack Type=Ignore Previous2024.12 | 8.33 | 68.75 | — | — | |
| Task ShieldModel=GPT-4o, Attack Type=Important Instructions2024.12 | 6.25 | 82.64 | — | — | |
| Task ShieldModel=GPT-4o2024.12 | 6.25 | — | 81.25 | 82.64 | |
| Task ShieldModel=GPT-4o-mini2024.12 | 6.25 | — | 43.75 | 37.5 | |
| Task ShieldModel=GPT-4o, Attack Type=Injecagent2024.12 | 4.17 | 77.78 | — | — | |
| Task ShieldModel=GPT-4o, Attack Type=Ignore Previous2024.12 | 1.39 | 72.22 | — | — | |
| PI DetectorModel=GPT-4o2024.12 | 0.69 | — | 43.75 | 31.25 |