Agentic Oversight on Agent-SafetyBench
84.06Detection AccuracyFORMALJUDGE
Evaluation results
| Method | Links | |
|---|---|---|
| FORMALJUDGEJudge Model=Claude-3.5-Sonnet2026.02 | 84.06 | |
| FORMALJUDGEJudge Model=Gemini-1.5-Pro2026.02 | 80.54 | |
| FORMALJUDGEJudge Model=GPT-4o2026.02 | 77.28 | |
| LLM + Struct.Judge Model=Gemini-1.5-Pro2026.02 | 76.6 | |
| FORMALJUDGEJudge Model=Qwen-7B2026.02 | 74.5 | |
| FORMALJUDGEJudge Model=Qwen-72B2026.02 | 73.26 | |
| LLM + Few-ShotJudge Model=Claude-3.5-Sonnet2026.02 | 72.73 | |
| LLMJudge Model=Claude-3.5-Sonnet2026.02 | 72.22 | |
| LLM + PythonJudge Model=Qwen-7B2026.02 | 72 | |
| FORMALJUDGEJudge Model=Qwen-32B2026.02 | 71.79 | |
| FORMALJUDGEJudge Model=Qwen-14B2026.02 | 71.77 | |
| LLM + CoTJudge Model=Claude-3.5-Sonnet2026.02 | 69.7 | |
| LLM + PythonJudge Model=GPT-4o2026.02 | 68.7 | |
| LLM + Struct.Judge Model=Claude-3.5-Sonnet2026.02 | 68.69 | |
| LLM + PythonJudge Model=Qwen-32B2026.02 | 68.18 | |
| LLM + PythonJudge Model=Gemini-1.5-Pro2026.02 | 67.38 | |
| LLM + CoTJudge Model=Gemini-1.5-Pro2026.02 | 67.33 | |
| LLM + PythonJudge Model=Qwen-14B2026.02 | 66.67 | |
| LLM + Struct.Judge Model=Qwen-14B2026.02 | 66.16 | |
| LLM + Few-ShotJudge Model=Gemini-1.5-Pro2026.02 | 65.63 | |
| LLM + PythonJudge Model=Qwen-72B2026.02 | 63 | |
| LLM + Struct.Judge Model=GPT-4o2026.02 | 62.3 | |
| LLM + Struct.Judge Model=Qwen-7B2026.02 | 62 | |
| LLMJudge Model=Qwen-7B2026.02 | 61.98 | |
| LLM + Few-ShotJudge Model=Qwen-7B2026.02 | 61.98 | |
| LLM + Struct.Judge Model=Qwen-32B2026.02 | 61.62 | |
| LLM + CoTJudge Model=GPT-4o2026.02 | 61.5 | |
| LLMJudge Model=Qwen-14B2026.02 | 60.1 | |
| LLM + Struct.Judge Model=Qwen-72B2026.02 | 59.5 | |
| LLM + Few-ShotJudge Model=GPT-4o2026.02 | 59.1 | |
| LLM + PythonJudge Model=Claude-3.5-Sonnet2026.02 | 58.59 | |
| LLMJudge Model=GPT-4o2026.02 | 58 | |
| LLMJudge Model=Gemini-1.5-Pro2026.02 | 58 | |
| LLMJudge Model=Qwen-72B2026.02 | 56.57 | |
| LLM + Few-ShotJudge Model=Qwen-14B2026.02 | 55.56 | |
| LLM + CoTJudge Model=Qwen-7B2026.02 | 55 | |
| LLM + CoTJudge Model=Qwen-14B2026.02 | 53.54 | |
| LLMJudge Model=Qwen-32B2026.02 | 52.53 | |
| LLM + CoTJudge Model=Qwen-32B2026.02 | 52.53 | |
| LLM + CoTJudge Model=Qwen-72B2026.02 | 52.5 | |
| LLM + Few-ShotJudge Model=Qwen-32B2026.02 | 52.02 | |
| LLM + Few-ShotJudge Model=Qwen-72B2026.02 | 51.5 |