Tool-use Agent Performance on τ²-bench
72.4ASRMemMorph
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MemMorphAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 72.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 68.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 64.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 60.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 57.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 56.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 55.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 53.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 52.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 51.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 51.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemMorphAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 50.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 46.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 45.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 45.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 40.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 38.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 38.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 35.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 32.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 31.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 30.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 29.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 28.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 27.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 27.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PoisonedRAGAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 26.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 25.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=GPT-4o, Poisoning Type=Direct2026.05 | 24.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 24.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 23.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 21.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 21 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 20.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 19.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 18.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 18.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Llama3 8B, Poisoning Type=Direct2026.05 | 18.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=GPT-4o mini, Poisoning Type=Direct2026.05 | 16.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Sonnet, Poisoning Type=Direct2026.05 | 16.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=GPT-3.5 20B, Poisoning Type=Direct2026.05 | 16.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 15 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Qwen2.5 7B, Poisoning Type=Direct2026.05 | 14.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 12.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Claude 3.5 Haiku, Poisoning Type=Direct2026.05 | 12.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAgent Backbone LLM=Qwen2.5 32B, Poisoning Type=Direct2026.05 | 11.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolHijackerAgent Backbone LLM=Llama3 70B, Poisoning Type=Direct2026.05 | 10.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ToolCommanderAgent Backbone LLM=Claude 3.5 Opus, Poisoning Type=Direct2026.05 | 7.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| A-MemModel=Qwen3-32B, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 53.12 | 38.75 | 38.12 | — | — | — | — | |
| A-MemModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 76.25 | 52.5 | 76.87 | — | — | — | — | |
| A-MemModel=GLM-4.6, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 76.88 | 70 | 68.75 | — | — | — | — | |
| AutoGuideModel=Qwen3-235B-A22B2026.01 | — | 32.2 | 21.1 | 14.9 | 49.07 | 51.12 | — | — | — | — | — | — | — | |
| AutoGuideModel=Claude 3.7 Sonnet2026.01 | — | 45.3 | 36.6 | 33.3 | 60.67 | 39.62 | — | — | — | — | — | — | — | |
| AWMModel=Qwen3-32B, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 55 | 40 | 38.12 | — | — | — | — | |
| AWMModel=Qwen3-32B, Experience extraction model alignment=GLM-4.6 used for experience extraction2026.04 | — | — | — | — | — | — | 57.5 | 41.25 | 40.62 | — | — | — | — | |
| AWMModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 76.25 | 53.75 | 77.5 | — | — | — | — | |
| AWMModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=GLM-4.6 used for experience extraction2026.04 | — | — | — | — | — | — | 76.25 | 53.75 | 77.5 | — | — | — | — | |
| AWMModel=GLM-4.6, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 77.5 | 71.25 | 70.63 | — | — | — | — | |
| ExpeLModel=Qwen3-32B, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 56.25 | 42.5 | 39.38 | — | — | — | — | |
| ExpeLModel=Qwen3-32B, Experience extraction model alignment=GLM-4.6 used for experience extraction2026.04 | — | — | — | — | — | — | 58.12 | 43.75 | 41.25 | — | — | — | — | |
| ExpeLModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 77.5 | 55.5 | 78.75 | — | — | — | — | |
| ExpeLModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=GLM-4.6 used for experience extraction2026.04 | — | — | — | — | — | — | 77.5 | 56.25 | 79.37 | — | — | — | — | |
| ExpeLModel=GLM-4.6, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 77.5 | 72.5 | 71.25 | — | — | — | — | |
| IRMAMethod category=Agentic context-engineering2026.06 | — | 23.4 | — | — | — | — | — | — | — | 9.6 | 53.1 | — | — | |
| LedgerMethod category=Agentic context-engineering2026.06 | — | 27.2 | — | — | — | — | — | — | — | 17.1 | 0 | — | — | |
| LLM-onlyModel=Qwen3-235B-A22B2026.01 | — | 39.5 | 22.2 | 13.2 | 55.37 | 43.84 | — | — | — | — | — | — | — | |
| LLM-onlyModel=Claude 3.7 Sonnet2026.01 | — | 46.2 | 33.9 | 27.2 | 63.29 | 36.92 | — | — | — | — | — | — | — | |
| No MemoryModel=Qwen3-32B, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 53.75 | 38.75 | 36.25 | — | — | — | — | |
| No MemoryModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 75.62 | 51.25 | 78.12 | — | — | — | — | |
| No MemoryModel=GLM-4.6, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 76.25 | 70 | 70.63 | — | — | — | — | |
| Off-policy Distill.Backbone=Qwen3-30B-A3B-Thinking-2507, Task Setup=Single-Task2026.06 | — | — | — | — | — | — | — | 60.5 | 92 | — | — | 41.7 | 74.7 | |
| Off-policy Distill.Backbone=Qwen3-30B-A3B-Thinking-2507, Task Setup=Multi-Task2026.06 | — | — | — | — | — | — | — | 61 | 88 | — | — | 41.5 | 65.8 | |
| Off. + On.Backbone=Qwen3-30B-A3B-Thinking-2507, Task Setup=Multi-Task2026.06 | — | — | — | — | — | — | — | 62 | 92.2 | — | — | 43.8 | 74.7 | |
| On-policy Distill.Backbone=Qwen3-30B-A3B-Thinking-2507, Task Setup=Multi-Task2026.06 | — | — | — | — | — | — | — | 61.1 | 88.8 | — | — | 41 | 72.2 | |
| Pre-RLBackbone=Qwen3-30B-A3B-Thinking-2507, Task Setup=Single-Task2026.06 | — | — | — | — | — | — | — | 52.9 | 77.4 | — | — | 30.9 | 51.1 | |
| ReActModel=Qwen3-235B-A22B2026.01 | — | 34.8 | 22.2 | 18.4 | 42.17 | 57.5 | — | — | — | — | — | — | — | |
| ReActModel=Claude 3.7 Sonnet2026.01 | — | 50.6 | 35.9 | 27.8 | 63.57 | 36.59 | — | — | — | — | — | — | — | |
| ReflexionModel=Qwen3-235B-A22B2026.01 | — | 34.5 | 21.6 | 16.7 | 54.56 | 45.66 | — | — | — | — | — | — | — | |
| ReflexionModel=Claude 3.7 Sonnet2026.01 | — | 53.5 | 43.3 | 37.7 | 65.4 | 34.65 | — | — | — | — | — | — | — | |
| REMEMBERERModel=Qwen3-235B-A22B2026.01 | — | 31.9 | 16.4 | 9.7 | 47.92 | 52.05 | — | — | — | — | — | — | — | |
| REMEMBERERModel=Claude 3.7 Sonnet2026.01 | — | 52.3 | 42.4 | 36.8 | 66.29 | 33.92 | — | — | — | — | — | — | — | |
| RLBackbone=Qwen3-30B-A3B-Thinking-2507, Task Setup=Single-Task2026.06 | — | — | — | — | — | — | — | 61.6 | 92.8 | — | — | 42.6 | 76.4 | |
| SkillXModel=Qwen3-32B, Experience extraction model alignment=GLM-4.6 used for experience extraction2026.04 | — | — | — | — | — | — | 66.87 | 47.5 | 43.75 | — | — | — | — | |
| SkillXModel=Kimi-K2-Instruct-0905, Experience extraction model alignment=GLM-4.6 used for experience extraction2026.04 | — | — | — | — | — | — | 78.12 | 58.75 | 82.5 | — | — | — | — | |
| SkillXModel=GLM-4.6, Experience extraction model alignment=Aligned with inference model2026.04 | — | — | — | — | — | — | 82.5 | 76.25 | 71.88 | — | — | — | — | |
| WISE-FlowModel=Qwen3-235B-A22B2026.01 | — | 39.5 | 25.2 | 19.3 | 63.43 | 36.65 | — | — | — | — | — | — | — | |
| WISE-FlowModel=Claude 3.7 Sonnet2026.01 | — | 56.4 | 46.8 | 42.1 | 69.36 | 30.75 | — | — | — | — | — | — | — |