Tool-use agent performance on MetaTool
85.9ASR (Success Rate)MemMorph
Evaluation Results
| Method | Links | |
|---|---|---|
| MemMorphAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 85.9 | |
| MemMorphAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 83.8 | |
| MemMorphAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 81.2 | |
| MemMorphAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 79.5 | |
| MemMorphAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 78.6 | |
| MemMorphAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 78.6 | |
| GCGAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 77.8 | |
| MemMorphAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 77.6 | |
| GCGAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 76.9 | |
| PoisonedRAGAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 75.2 | |
| MemMorphAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 72.7 | |
| GCGAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 72.6 | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 72.6 | |
| MemMorphAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 72.6 | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 70.9 | |
| GCGAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 70.1 | |
| MemMorphAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 70.1 | |
| PoisonedRAGAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 68.4 | |
| GCGAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 67.8 | |
| PoisonedRAGAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 63.2 | |
| PoisonedRAGAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 62.4 | |
| GCGAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 58.1 | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 57.3 | |
| ToolHijackerAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 57.2 | |
| ToolCommanderAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 56.3 | |
| PoisonedRAGAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 55.6 | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 55.6 | |
| ToolCommanderAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 55.6 | |
| GCGAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 54.7 | |
| ToolHijackerAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 54.4 | |
| GCGAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 53 | |
| GCGAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 53 | |
| PoisonedRAGAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 50.4 | |
| GCGAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 49.6 | |
| PoisonedRAGAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 45.3 | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 44.4 | |
| ToolCommanderAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 44.3 | |
| ToolHijackerAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 44.1 | |
| ToolHijackerAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 41.9 | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 35.8 | |
| ToolHijackerAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 32.1 | |
| ToolHijackerAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 30.7 | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 25.8 | |
| ToolHijackerAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 25.6 | |
| ToolCommanderAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 24.8 | |
| ToolCommanderAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 23 | |
| ToolCommanderAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 16.2 | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 15.4 | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 11.9 | |
| ToolCommanderAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 8.5 |