Agentic Safety and Utility Evaluation on PowerSeeking Bench
1Safety ScoreQwen3Guard-8B
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Qwen3Guard-8BAgent=GPT-4o-mini2026.06 | 1 | 0.05 | 0.33 | |
| Qwen3Guard-8BAgent=Gemini-2.5-flash2026.06 | 1 | 0.07 | 0.34 | |
| Qwen3Guard-8BAgent=Llama-3.1-8B2026.06 | 1 | 0 | 0.29 | |
| SafeMCPAgent=Gemini-2.5-flash2026.06 | 0.97 | 0.29 | 0.5 | |
| Lakera-ChainGuardAgent=Llama-3.1-8B2026.06 | 0.96 | 0.06 | 0.34 | |
| Lakera-ChainGuardAgent=Gemini-2.5-flash2026.06 | 0.95 | 0.22 | 0.45 | |
| Lakera-ChainGuardAgent=GPT-4o-mini2026.06 | 0.94 | 0.11 | 0.37 | |
| SafeMCPAgent=GPT-4o-mini2026.06 | 0.92 | 0.35 | 0.54 | |
| SafeMCPAgent=Llama-3.1-8B2026.06 | 0.88 | 0.19 | 0.42 | |
| NeMoGuard-8BAgent=Gemini-2.5-flash2026.06 | 0.84 | 0.22 | 0.44 | |
| Llama Guard 3Agent=GPT-4o-mini2026.06 | 0.78 | 0.13 | 0.37 | |
| NeMoGuard-8BAgent=Llama-3.1-8B2026.06 | 0.77 | 0.08 | 0.33 | |
| Llama Guard 3Agent=Gemini-2.5-flash2026.06 | 0.76 | 0.09 | 0.33 | |
| NeMoGuard-8BAgent=GPT-4o-mini2026.06 | 0.75 | 0.12 | 0.35 | |
| AgentMonitorAgent=GPT-4o-mini2026.06 | 0.75 | 0.22 | 0.42 | |
| AgentMonitorAgent=Gemini-2.5-flash2026.06 | 0.74 | 0.2 | 0.41 | |
| AgentMonitorAgent=Llama-3.1-8B2026.06 | 0.73 | 0.19 | 0.4 | |
| SafironAgent=Gemini-2.5-flash2026.06 | 0.7 | 0.12 | 0.34 | |
| Llama Guard 3Agent=Llama-3.1-8B2026.06 | 0.69 | 0.04 | 0.28 | |
| SafironAgent=GPT-4o-mini2026.06 | 0.67 | 0.16 | 0.36 | |
| SafironAgent=Llama-3.1-8B2026.06 | 0.6 | 0.06 | 0.28 | |
| w/o moderatorAgent=Gemini-2.5-flash2026.06 | 0.57 | 0.12 | 0.31 | |
| w/o moderatorAgent=Llama-3.1-8B2026.06 | 0.5 | 0.13 | 0.29 | |
| w/o moderatorAgent=GPT-4o-mini2026.06 | 0.37 | 0.17 | 0.26 |