Agentic Oversight on VitaBench
82.13Detection AccuracyFORMALJUDGE
Evaluation results
| Method | Links | |
|---|---|---|
| FORMALJUDGEJudge Model=Gemini-1.5-Pro2026.02 | 82.13 | |
| FORMALJUDGEJudge Model=GPT-4o2026.02 | 78.53 | |
| FORMALJUDGEJudge Model=Qwen-32B2026.02 | 78.53 | |
| LLM + PythonJudge Model=Gemini-1.5-Pro2026.02 | 76.23 | |
| LLM + Struct.Judge Model=Gemini-1.5-Pro2026.02 | 75.04 | |
| FORMALJUDGEJudge Model=Claude-3.5-Sonnet2026.02 | 74.93 | |
| LLM + Struct.Judge Model=GPT-4o2026.02 | 74.72 | |
| FORMALJUDGEJudge Model=Qwen-14B2026.02 | 74.52 | |
| LLMJudge Model=Gemini-1.5-Pro2026.02 | 74.36 | |
| LLM + PythonJudge Model=Qwen-32B2026.02 | 73.85 | |
| LLM + Few-ShotJudge Model=Gemini-1.5-Pro2026.02 | 73.54 | |
| LLM + PythonJudge Model=GPT-4o2026.02 | 73.14 | |
| LLMJudge Model=GPT-4o2026.02 | 72.23 | |
| LLM + Few-ShotJudge Model=GPT-4o2026.02 | 72.21 | |
| LLM + CoTJudge Model=Gemini-1.5-Pro2026.02 | 72.15 | |
| FORMALJUDGEJudge Model=Qwen-7B2026.02 | 70.97 | |
| LLM + Struct.Judge Model=Qwen-32B2026.02 | 70.82 | |
| LLM + PythonJudge Model=Claude-3.5-Sonnet2026.02 | 69.84 | |
| LLM + CoTJudge Model=Claude-3.5-Sonnet2026.02 | 68.42 | |
| LLM + Few-ShotJudge Model=Qwen-32B2026.02 | 67.93 | |
| LLMJudge Model=Qwen-14B2026.02 | 67.82 | |
| LLM + PythonJudge Model=Qwen-14B2026.02 | 67.51 | |
| LLM + CoTJudge Model=Qwen-32B2026.02 | 67.02 | |
| LLM + CoTJudge Model=Qwen-14B2026.02 | 66.13 | |
| LLMJudge Model=Qwen-32B2026.02 | 66.05 | |
| LLM + PythonJudge Model=Qwen-7B2026.02 | 65.92 | |
| LLM + CoTJudge Model=GPT-4o2026.02 | 65.84 | |
| LLM + Struct.Judge Model=Qwen-7B2026.02 | 64.52 | |
| LLM + Struct.Judge Model=Qwen-14B2026.02 | 64.45 | |
| LLM + Few-ShotJudge Model=Qwen-14B2026.02 | 64.43 | |
| FORMALJUDGEJudge Model=Qwen-72B2026.02 | 64.24 | |
| LLM + CoTJudge Model=Qwen-7B2026.02 | 62.76 | |
| LLMJudge Model=Claude-3.5-Sonnet2026.02 | 62.61 | |
| LLM + PythonJudge Model=Qwen-72B2026.02 | 62.14 | |
| LLM + Struct.Judge Model=Claude-3.5-Sonnet2026.02 | 62.03 | |
| LLMJudge Model=Qwen-7B2026.02 | 60.94 | |
| LLM + Few-ShotJudge Model=Qwen-7B2026.02 | 60.91 | |
| LLMJudge Model=Qwen-72B2026.02 | 59.31 | |
| LLM + Struct.Judge Model=Qwen-72B2026.02 | 59.28 | |
| LLM + Few-ShotJudge Model=Claude-3.5-Sonnet2026.02 | 58.52 | |
| LLM + CoTJudge Model=Qwen-72B2026.02 | 58.04 | |
| LLM + Few-ShotJudge Model=Qwen-72B2026.02 | 56.83 |