LLM Agent Utility on AgentHarm Benign Requests
69.1Utility ScoreBaseline (No Defense)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Baseline (No Defense)Model=Mistral-8B2026.05 | 69.1 | 35.8 | 0 | 69.1 | |
| Rule TraverseModel=Mistral-8B2026.05 | 65.7 | 28.4 | 1.1 | 66.4 | |
| SAFEHARBOR (Base)Model=GPT-4o2026.05 | 63.6 | 42.6 | 25 | 84.5 | |
| RAGModel=Mistral-8B2026.05 | 62.8 | 23.3 | 0 | 62.8 | |
| A-MemModel=Mistral-8B2026.05 | 62.3 | 28.4 | 0 | 62.3 | |
| A-MemModel=GPT-4o2026.05 | 61.3 | 40.3 | 9.1 | 67.4 | |
| AgentAlign (SFT)Model=Mistral-8B2026.05 | 55.3 | 26.1 | 2.3 | 56.6 | |
| SAFEHARBOR (Base)Model=Mistral-8B2026.05 | 53 | 25 | 21 | 66.2 | |
| Baseline (No Defense)Model=Qwen2.5-7B2026.05 | 52.8 | 13.6 | 0 | 52.8 | |
| LlamaGuardModel=GPT-4o2026.05 | 52.4 | 37.5 | 29 | 73.4 | |
| LlamaGuardModel=Mistral-8B2026.05 | 52.3 | 25.6 | 22.7 | 67.7 | |
| Rule TraverseModel=Qwen2.5-7B2026.05 | 49.4 | 13.6 | 4.5 | 51.8 | |
| SAFEHARBOR (Base)Model=Qwen2.5-7B2026.05 | 49.4 | 14.8 | 9.1 | 53.8 | |
| A-MemModel=Qwen2.5-7B2026.05 | 46.7 | 14.2 | 13.6 | 46.8 | |
| Baseline (No Defense)Model=GPT-4o2026.05 | 44.2 | 29.5 | 50 | 81.4 | |
| RAGModel=Qwen2.5-7B2026.05 | 43.3 | 12.5 | 10.2 | 44.6 | |
| RAGModel=GPT-4o2026.05 | 42.2 | 29 | 55.7 | 82.9 | |
| LlamaGuardModel=Qwen2.5-7B2026.05 | 40.8 | 11.9 | 22.7 | 52.8 | |
| GuardAgentModel=Mistral-8B2026.05 | 35.1 | 12.5 | 58.5 | 51.6 | |
| GuardAgentModel=Qwen2.5-7B2026.05 | 33.4 | 6.3 | 44 | 43.1 | |
| GuardAgentModel=GPT-4o2026.05 | 24.6 | 13.6 | 50 | 43.3 | |
| AgentAlign (SFT)Model=Qwen2.5-7B2026.05 | 20.7 | 0.6 | 11.9 | 23.1 | |
| Rule TraverseModel=GPT-4o2026.05 | 12.1 | 5.1 | 88.1 | 68.1 |