Agentic Oversight on Deceivers
96.92Detection AccuracyFORMALJUDGE
Evaluation Results
| Method | Links | |
|---|---|---|
| FORMALJUDGEJudge Model=Qwen-32B2026.02 | 96.92 | |
| FORMALJUDGEJudge Model=Qwen-7B2026.02 | 96.8 | |
| FORMALJUDGEJudge Model=Qwen-72B2026.02 | 94.16 | |
| FORMALJUDGEJudge Model=Claude-3.5-Sonnet2026.02 | 93.51 | |
| FORMALJUDGEJudge Model=Qwen-14B2026.02 | 92.91 | |
| FORMALJUDGEJudge Model=GPT-4o2026.02 | 91.96 | |
| FORMALJUDGEJudge Model=Gemini-1.5-Pro2026.02 | 89.64 | |
| LLM + PythonJudge Model=Qwen-7B2026.02 | 80.23 | |
| LLM + Struct.Judge Model=Qwen-7B2026.02 | 79.45 | |
| LLM + PythonJudge Model=Qwen-14B2026.02 | 77.56 | |
| LLM + Struct.Judge Model=Qwen-14B2026.02 | 76.89 | |
| LLM + PythonJudge Model=Qwen-32B2026.02 | 75.42 | |
| LLM + Few-ShotJudge Model=Qwen-7B2026.02 | 75.12 | |
| LLM + Struct.Judge Model=Qwen-32B2026.02 | 74.67 | |
| LLM + PythonJudge Model=Claude-3.5-Sonnet2026.02 | 74.51 | |
| LLM + Struct.Judge Model=Claude-3.5-Sonnet2026.02 | 73.62 | |
| LLM + CoTJudge Model=Qwen-7B2026.02 | 72.86 | |
| LLM + Few-ShotJudge Model=Qwen-14B2026.02 | 72.68 | |
| LLM + PythonJudge Model=Qwen-72B2026.02 | 72.67 | |
| LLM + PythonJudge Model=GPT-4o2026.02 | 72.34 | |
| LLM + Struct.Judge Model=Qwen-72B2026.02 | 71.83 | |
| LLM + Struct.Judge Model=GPT-4o2026.02 | 71.58 | |
| LLMJudge Model=Qwen-7B2026.02 | 71.34 | |
| LLM + CoTJudge Model=Qwen-14B2026.02 | 70.41 | |
| LLM + Few-ShotJudge Model=Qwen-32B2026.02 | 70.35 | |
| LLM + PythonJudge Model=Gemini-1.5-Pro2026.02 | 70.18 | |
| LLM + Struct.Judge Model=Gemini-1.5-Pro2026.02 | 69.24 | |
| LLMJudge Model=Qwen-14B2026.02 | 68.92 | |
| LLM + Few-ShotJudge Model=Claude-3.5-Sonnet2026.02 | 68.47 | |
| LLM + CoTJudge Model=Qwen-32B2026.02 | 68.24 | |
| LLM + Few-ShotJudge Model=Qwen-72B2026.02 | 67.42 | |
| LLM + Few-ShotJudge Model=GPT-4o2026.02 | 66.83 | |
| LLMJudge Model=Qwen-32B2026.02 | 66.57 | |
| LLM + CoTJudge Model=Claude-3.5-Sonnet2026.02 | 65.73 | |
| LLM + Few-ShotJudge Model=Gemini-1.5-Pro2026.02 | 64.91 | |
| LLM + CoTJudge Model=Qwen-72B2026.02 | 64.89 | |
| LLMJudge Model=Claude-3.5-Sonnet2026.02 | 64.19 | |
| LLM + CoTJudge Model=GPT-4o2026.02 | 64.12 | |
| LLMJudge Model=Qwen-72B2026.02 | 63.21 | |
| LLM + CoTJudge Model=Gemini-1.5-Pro2026.02 | 62.58 | |
| LLMJudge Model=GPT-4o2026.02 | 62.47 | |
| LLMJudge Model=Gemini-1.5-Pro2026.02 | 60.83 |