Agent Safety Evaluation on AgentHarm Benign Requests
79Safety ScoreGPT-4o
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-4oBase Model=GPT-4o2026.06 | 79 | 54 | 0 | |
| SafironBase Model=GPT-4o2026.06 | 79 | 54 | 3 | |
| RL-GuardBase Model=GPT-4o2026.06 | 76 | 48 | 1 | |
| SafeMCPBase Model=GPT-4o2026.06 | 76 | 53 | 1 | |
| GPT-4o-miniBase Model=GPT-4o-mini2026.06 | 71 | 39 | 0 | |
| SafironBase Model=GPT-4o-mini2026.06 | 71 | 39 | 3 | |
| SafeMCPBase Model=GPT-4o-mini2026.06 | 69 | 36 | 1 | |
| RL-GuardBase Model=GPT-4o-mini2026.06 | 68 | 40 | 7 | |
| Llama Guard 3Base Model=GPT-4o2026.06 | 60 | 40 | 31 | |
| Llama Guard 3Base Model=GPT-4o-mini2026.06 | 58 | 35 | 31 | |
| LLaMA-3.1-Instruct-8BBase Model=LLaMA-3.1-Instruct-8B2026.06 | 55 | 11 | 0 | |
| NeMoGuard-8bBase Model=GPT-4o2026.06 | 53 | 35 | 38 | |
| SafironBase Model=LLaMA-3.1-Instruct-8B2026.06 | 53 | 13 | 3 | |
| SafeMCPBase Model=LLaMA-3.1-Instruct-8B2026.06 | 52 | 9 | 0 | |
| NeMoGuard-8bBase Model=GPT-4o-mini2026.06 | 50 | 30 | 36 | |
| RL-GuardBase Model=LLaMA-3.1-Instruct-8B2026.06 | 49 | 6 | 1 | |
| Llama Guard 3Base Model=LLaMA-3.1-Instruct-8B2026.06 | 41 | 8 | 25 | |
| NeMoGuard-8bBase Model=LLaMA-3.1-Instruct-8B2026.06 | 41 | 8 | 27 | |
| AgentMonitorBase Model=GPT-4o2026.06 | 36 | 27 | 56 | |
| AgentMonitorBase Model=GPT-4o-mini2026.06 | 36 | 23 | 57 | |
| Qwen3Guard-8BBase Model=GPT-4o2026.06 | 29 | 21 | 68 | |
| Qwen3Guard-8BBase Model=GPT-4o-mini2026.06 | 27 | 19 | 68 | |
| AgentMonitorBase Model=LLaMA-3.1-Instruct-8B2026.06 | 19 | 4 | 63 | |
| Qwen3Guard-8BBase Model=LLaMA-3.1-Instruct-8B2026.06 | 17 | 3 | 67 | |
| Lakera-ChainGuardBase Model=GPT-4o2026.06 | 15 | 7 | 99 | |
| Lakera-ChainGuardBase Model=GPT-4o-mini2026.06 | 14 | 5 | 99 | |
| Lakera-ChainGuardBase Model=LLaMA-3.1-Instruct-8B2026.06 | 9 | 1 | 85 |