Agent Safety Evaluation on AgentHarm Harmful Requests
59ScoreGPT-4o-mini
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-4o-miniBase Model=GPT-4o-mini2026.06 | 59 | 28 | 6 | |
| SafironBase Model=GPT-4o-mini2026.06 | 52 | 28 | 6 | |
| GPT-4oBase Model=GPT-4o2026.06 | 51 | 34 | 5 | |
| LLaMA-3.1-Instruct-8BBase Model=LLaMA-3.1-Instruct-8B2026.06 | 47 | 9 | 6 | |
| SafironBase Model=GPT-4o2026.06 | 45 | 33 | 6 | |
| RL-GuardBase Model=LLaMA-3.1-Instruct-8B2026.06 | 42.8 | 7 | 5 | |
| SafironBase Model=LLaMA-3.1-Instruct-8B2026.06 | 39 | 9 | 9 | |
| RL-GuardBase Model=GPT-4o2026.06 | 36 | 26 | 9 | |
| RL-GuardBase Model=GPT-4o-mini2026.06 | 36 | 24 | 10 | |
| SafeMCPBase Model=GPT-4o-mini2026.06 | 19 | 5 | 9 | |
| SafeMCPBase Model=LLaMA-3.1-Instruct-8B2026.06 | 18 | 0 | 8 | |
| SafeMCPBase Model=GPT-4o2026.06 | 17 | 5 | 9 | |
| NeMoGuard-8bBase Model=LLaMA-3.1-Instruct-8B2026.06 | 13 | 3 | 14 | |
| NeMoGuard-8bBase Model=GPT-4o2026.06 | 12 | 9 | 13 | |
| NeMoGuard-8bBase Model=GPT-4o-mini2026.06 | 10 | 6 | 14 | |
| AgentMonitorBase Model=GPT-4o-mini2026.06 | 8 | 5 | 16 | |
| Lakera-ChainGuardBase Model=LLaMA-3.1-Instruct-8B2026.06 | 8 | 2 | 16 | |
| AgentMonitorBase Model=GPT-4o2026.06 | 7 | 5 | 16 | |
| AgentMonitorBase Model=LLaMA-3.1-Instruct-8B2026.06 | 6 | 2 | 17 | |
| Llama Guard 3Base Model=GPT-4o2026.06 | 2 | 1 | 18 | |
| Lakera-ChainGuardBase Model=GPT-4o2026.06 | 2 | 2 | 18 | |
| Llama Guard 3Base Model=GPT-4o-mini2026.06 | 2 | 1 | 18 | |
| Llama Guard 3Base Model=LLaMA-3.1-Instruct-8B2026.06 | 2 | 0 | 18 | |
| Lakera-ChainGuardBase Model=GPT-4o-mini2026.06 | 1 | 0 | 18 | |
| Qwen3Guard-8BBase Model=GPT-4o2026.06 | 0 | 0 | 18 | |
| Qwen3Guard-8BBase Model=GPT-4o-mini2026.06 | 0 | 0 | 18 | |
| Qwen3Guard-8BBase Model=LLaMA-3.1-Instruct-8B2026.06 | 0 | 0 | 18 |