Malicious behavior measurement on AgentHarm Harmful
0Harm RateRule Traverse
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Rule TraverseModel=GPT-4o2026.05 | 0 | 100 | 0 | 0 | |
| AgentAlign (SFT)Model=Qwen2.5-7B2026.05 | 0 | 90.3 | 14.6 | 1.4 | |
| LlamaGuardModel=Qwen2.5-7B2026.05 | 0.6 | 96 | 46.1 | 1.8 | |
| SAFEHARBOR (Base)Model=Qwen2.5-7B2026.05 | 0.6 | 89.2 | 35.8 | 3.9 | |
| LlamaGuardModel=Mistral-8B2026.05 | 1.1 | 96.6 | 70 | 2.4 | |
| AgentAlign (SFT)Model=Mistral-8B2026.05 | 1.7 | 82.4 | 53.7 | 9.5 | |
| LlamaGuardModel=GPT-4o2026.05 | 2.3 | 95.5 | 68.8 | 3.1 | |
| GuardAgentModel=GPT-4o2026.05 | 2.8 | 94.9 | 75.3 | 11 | |
| SAFEHARBOR (Base)Model=Mistral-8B2026.05 | 2.8 | 86.9 | 50.4 | 6.6 | |
| GuardAgentModel=Qwen2.5-7B2026.05 | 3.4 | 81.3 | 37.9 | 16 | |
| SAFEHARBOR (Base)Model=GPT-4o2026.05 | 5.1 | 93.2 | 86.8 | 6.3 | |
| Rule TraverseModel=Qwen2.5-7B2026.05 | 5.1 | 75.6 | 50.8 | 12.4 | |
| RAGModel=Qwen2.5-7B2026.05 | 5.1 | 51.7 | 34.6 | 21.4 | |
| GPT-5Safety Scaffolding=MOSAIC, Evaluation Mode=Zero-shot2026.03 | 6 | 91 | 57 | — | |
| Phi-4Safety Scaffolding=Base, Evaluation Mode=Zero-shot2026.03 | 6 | 94 | 68 | — | |
| GuardAgentModel=Mistral-8B2026.05 | 6.8 | 93.2 | 75.6 | 14.1 | |
| GPT-4oSafety Scaffolding=MOSAIC, Evaluation Mode=Zero-shot2026.03 | 7 | 92 | 67 | — | |
| A-MemModel=Qwen2.5-7B2026.05 | 7.4 | 64.8 | 42.7 | 23.4 | |
| Qwen3-4B-ThinkSafety Scaffolding=MOSAIC, Evaluation Mode=Zero-shot2026.03 | 8 | 89 | 62 | — | |
| RAGModel=GPT-4o2026.05 | 8 | 89.8 | 85.6 | 9.1 | |
| A-MemModel=GPT-4o2026.05 | 8 | 86.9 | 84.9 | 11.1 | |
| Qwen2.5-7BSafety Scaffolding=MOSAIC, Evaluation Mode=Zero-shot2026.03 | 9 | 87 | 52 | — | |
| Qwen3-4B-ThinkSafety Scaffolding=Base, Evaluation Mode=Zero-shot2026.03 | 9 | 86 | 59 | — | |
| Phi-4Safety Scaffolding=MOSAIC, Evaluation Mode=Zero-shot2026.03 | 9 | 88 | 63 | — | |
| GPT-5Safety Scaffolding=Base, Evaluation Mode=Zero-shot2026.03 | 11 | 0 | 11 | — | |
| Rule TraverseModel=Mistral-8B2026.05 | 12.5 | 56.3 | 66.6 | 29.2 | |
| Baseline (No Defense)Model=Qwen2.5-7B2026.05 | 14.2 | 21.6 | 52.4 | 41.9 | |
| Qwen2.5-7BSafety Scaffolding=Base, Evaluation Mode=Zero-shot2026.03 | 18 | 74 | 58 | — | |
| RAGModel=Mistral-8B2026.05 | 23.9 | 1.1 | 59.8 | 59.6 | |
| Baseline (No Defense)Model=GPT-4o2026.05 | 25 | 58 | 88.9 | 38.1 | |
| Baseline (No Defense)Model=Mistral-8B2026.05 | 27.8 | 0 | 67.4 | 67.4 | |
| A-MemModel=Mistral-8B2026.05 | 29.1 | 1.2 | 63.6 | 63.3 | |
| GPT-4oSafety Scaffolding=Base, Evaluation Mode=Zero-shot2026.03 | 31 | 0 | 31 | — |