Policy Violation Detection on PhantomPolicy original (violation-ground-truth)
54Violated CountClaude Sonnet 4.6
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Claude Sonnet 4.6Condition=Execution-oriented baseline, Labels=Human-reviewed trace labels2026.04 | 54 | 6 | |
| Claude Opus 4.6Condition=Execution-oriented baseline, Labels=Human-reviewed trace labels2026.04 | 56 | 4 | |
| GPT-5.4Condition=Execution-oriented baseline, Labels=Human-reviewed trace labels2026.04 | 58 | 2 | |
| GPT-5 miniCondition=Execution-oriented baseline, Labels=Human-reviewed trace labels2026.04 | 59 | 1 | |
| GPT-5.4 nanoCondition=Execution-oriented baseline, Labels=Human-reviewed trace labels2026.04 | 59 | 1 |