Tool-use on ToolBench (Avg token length, TSR, ASR)
99.61Average Success Rate (ASR)Llama 3.3-70B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Llama 3.3-70BAttack Scenario=overthink2026.01 | 99.61 | 389 | — | |
| Llama 3.3-70BAttack Scenario=MCTS-based tool-layer DoS attack2026.01 | 96.17 | 81,830 | — | |
| Llama-DeepSeekAttack Scenario=overthink2026.01 | 91.41 | 725 | — | |
| MemMorphAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 84.7 | — | — | |
| Seed 32BAttack Scenario=MCTS-based tool-layer DoS attack2026.01 | 84.29 | 85,037 | — | |
| Seed 32BAttack Scenario=overthink2026.01 | 83.56 | 4,223 | — | |
| GLM 4.5-AirAttack Scenario=MCTS-based tool-layer DoS attack2026.01 | 83.14 | 63,694 | — | |
| MemMorphAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 81.6 | — | — | |
| Mistral LargeAttack Scenario=MCTS-based tool-layer DoS attack2026.01 | 81.23 | 61,354 | — | |
| MemMorphAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 80.8 | — | — | |
| Qwen 3-32BAttack Scenario=MCTS-based tool-layer DoS attack2026.01 | 80.46 | 65,273 | — | |
| Qwen 3-32BAttack Scenario=overthink2026.01 | 80.08 | 8,743 | — | |
| MemMorphAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 79.6 | — | — | |
| Mistral LargeAttack Scenario=overthink2026.01 | 79.3 | 1,453 | — | |
| Llama-DeepSeekAttack Scenario=MCTS-based tool-layer DoS attack2026.01 | 78.93 | 65,546 | — | |
| PoisonedRAGAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 78 | — | — | |
| MemMorphAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 77.6 | — | — | |
| MemMorphAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 74.7 | — | — | |
| PoisonedRAGAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 72.4 | — | — | |
| PoisonedRAGAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 71.2 | — | — | |
| MemMorphAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 70.4 | — | — | |
| MemMorphAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 68.2 | — | — | |
| ToolCommanderAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 66.3 | — | — | |
| MemMorphAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 66.1 | — | — | |
| GCGAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 65.4 | — | — | |
| GCGAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 64.8 | — | — | |
| MemMorphAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 64.3 | — | — | |
| GCGAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 63.3 | — | — | |
| GCGAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 62.9 | — | — | |
| PoisonedRAGAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 58.4 | — | — | |
| GLM 4.5-AirAttack Scenario=overthink2026.01 | 57.54 | 12,580 | — | |
| ToolHijackerAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 55.2 | — | — | |
| GCGAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 55.1 | — | — | |
| GCGAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 53.1 | — | — | |
| GCGAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 47.1 | — | — | |
| ToolCommanderAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 46.9 | — | — | |
| ToolCommanderAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 44.5 | — | — | |
| PoisonedRAGAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 35.7 | — | — | |
| ToolCommanderAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 35.7 | — | — | |
| PoisonedRAGAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 33.3 | — | — | |
| GCGAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 31.6 | — | — | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 29.6 | — | — | |
| ToolHijackerAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 27.6 | — | — | |
| ToolHijackerAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 27.3 | — | — | |
| GCGAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 26.6 | — | — | |
| ToolCommanderAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 26.1 | — | — | |
| PoisonedRAGAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 23.7 | — | — | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 21.4 | — | — | |
| ToolHijackerAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 21.1 | — | — | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 20.4 | — | — | |
| ToolCommanderAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 20.4 | — | — | |
| GCGAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 19.4 | — | — | |
| ToolCommanderAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 18.3 | — | — | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 17.1 | — | — | |
| ToolHijackerAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 16.3 | — | — | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 16.2 | — | — | |
| ToolHijackerAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 15.7 | — | — | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 12.9 | — | — | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 9.2 | — | — | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 8.1 | — | — | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 4.1 | — | — | |
| ToolHijackerAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 3.1 | — | — | |
| GLM 4.5-AirAttack Scenario=under no attack2026.01 | — | 634 | 95.02 | |
| Llama 3.3-70BAttack Scenario=under no attack2026.01 | — | 260 | 98.08 | |
| Llama-DeepSeekAttack Scenario=under no attack2026.01 | — | 197 | 86.59 | |
| Mistral LargeAttack Scenario=under no attack2026.01 | — | 127 | 90.8 | |
| Qwen 3-32BAttack Scenario=under no attack2026.01 | — | 638 | 94.64 | |
| Seed 32BAttack Scenario=under no attack2026.01 | — | 1,298 | 90.42 |