Policy Compliance Evaluation on SOCpilot 1.0 (test)
0Violation RateHuman baseline
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Human baseline2026.06 | 0 | 0 | 100 | 0 | |
| ClaudePrompting=zero-shot2026.06 | 36 | 30 | 75.67 | 6.78 | |
| GPTPrompting=policy-augmented2026.06 | 47 | 40 | 74 | 9.17 | |
| GPTPrompting=zero-shot2026.06 | 54 | 47 | 81 | 10.92 | |
| ClaudePrompting=policy-augmented2026.06 | 87 | 82 | 76.34 | 15.44 |