Policy Following on POLICYSHIFTBENCH (Adaptive split)
89AccuracyHuman Performance
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Human Performance2026.07 | 89 | 90.1 | — | |
| Human Performance2026.07 | 88 | 89.1 | — | |
| PolicyShiftGuard-3BTime (ms)=128.52026.07 | 86.6 | 85.6 | 79 | |
| PolicyShiftGuard-7BTime (ms)=163.92026.07 | 86.3 | 86.8 | 73.8 | |
| PolicyShiftGuard-7BTime (ms)=163.92026.07 | 78.1 | 76.9 | 72.1 | |
| Gemini-3-Flash-PreviewTime (ms)=5963.52026.07 | 74.2 | 70.6 | 50.6 | |
| PolicyShiftGuard-3BTime (ms)=128.52026.07 | 73.8 | 66.7 | 64.8 | |
| Qwen3.5-35B-A3BThinking mode=true, Time (ms)=13099.42026.07 | 71.3 | 63.5 | 36.2 | |
| Gemini-3-Flash-PreviewTime (ms)=5963.52026.07 | 70.1 | 67 | 45.4 | |
| GPT-5.4Time (ms)=1037.52026.07 | 67.8 | 57.8 | 45.5 | |
| Qwen3.5-35B-A3BThinking mode=true, Time (ms)=13099.42026.07 | 67.2 | 52.4 | 29.8 | |
| GPT-5.4Time (ms)=1037.52026.07 | 66.6 | 57.8 | 42.7 | |
| Qwen3.5-2BTime (ms)=57.32026.07 | 66.4 | 18.2 | 0 | |
| Qwen2.5-VL-72BTime (ms)=540.32026.07 | 66.2 | 61.7 | 32.1 | |
| Qwen3.5-35B-A3BThinking mode=false, Time (ms)=118.52026.07 | 65.8 | 63.7 | 25.4 | |
| Qwen3.5-4BThinking mode=true, Time (ms)=20292.22026.07 | 64.3 | 37.9 | 26.3 | |
| Qwen3.5-4BThinking mode=true, Time (ms)=20292.22026.07 | 62.8 | 31.6 | 17.8 | |
| Qwen2.5-VL-72BTime (ms)=540.32026.07 | 62.1 | 49.4 | 27.4 | |
| Claude-Sonnet-4.6Time (ms)=1065.72026.07 | 61.1 | 52.5 | 25.7 | |
| Qwen3.5-35B-A3BThinking mode=false, Time (ms)=118.52026.07 | 60.2 | 48.1 | 18.6 | |
| Qwen2.5-VL-32BTime (ms)=358.42026.07 | 59.9 | 59.2 | 16.1 | |
| Qwen3.5-2BTime (ms)=57.32026.07 | 57.7 | 9.1 | 0 | |
| Claude-Sonnet-4.6Time (ms)=1065.72026.07 | 57.3 | 39.8 | 18.8 | |
| Qwen2.5-VL-32BTime (ms)=358.42026.07 | 56.5 | 47.2 | 9.4 | |
| SafeGuard-VL-RL-7BTime (ms)=154.42026.07 | 55.8 | 62.5 | 6.3 | |
| Qwen2.5-VL-7BTime (ms)=273.32026.07 | 54.5 | 29 | 9.5 | |
| SafeGuard-VL-RL-7BTime (ms)=154.42026.07 | 53.4 | 51 | 4 | |
| Qwen3.5-4BThinking mode=false, Time (ms)=89.22026.07 | 52.9 | 59 | 3 | |
| Qwen2.5-VL-7BTime (ms)=273.32026.07 | 52.6 | 20.6 | 4.8 | |
| Qwen3.5-4BThinking mode=false, Time (ms)=89.22026.07 | 52.3 | 47.4 | 4.4 | |
| GuardReasoner-VL-3BTime (ms)=2084.52026.07 | 52.2 | 59.2 | 3.2 | |
| GuardReasoner-VL-3BTime (ms)=2084.52026.07 | 52.1 | 63.6 | 4.2 | |
| Llama Guard-4-12BTime (ms)=277.92026.07 | 52 | 29.2 | 2 | |
| GuardReasoner-VL-7BTime (ms)=3300.32026.07 | 51.6 | 61.5 | 10.8 | |
| GuardReasoner-VL-7BTime (ms)=3300.32026.07 | 51.6 | 57.9 | 10.5 | |
| ShieldGemma2-4BTime (ms)=134.02026.07 | 51.4 | 26.8 | 4.8 | |
| Llama Guard-4-12BTime (ms)=277.92026.07 | 51.4 | 19.6 | 1.6 | |
| Qwen2.5-VL-3BTime (ms)=231.12026.07 | 50.9 | 61 | 2.4 | |
| ShieldGemma2-4BTime (ms)=134.02026.07 | 50.8 | 36.1 | 4.5 | |
| Qwen3.5-0.8BTime (ms)=42.92026.07 | 50.7 | 3.5 | 2.4 | |
| Qwen3.5-0.8BTime (ms)=42.92026.07 | 50.7 | 3.9 | 1.8 | |
| Qwen2.5-VL-3BTime (ms)=231.12026.07 | 50 | 65.9 | 2 | |
| QwenGuard-7BTime (ms)=211.62026.07 | 47.2 | 39.2 | 9.6 | |
| QwenGuard-7BTime (ms)=211.62026.07 | 44.6 | 39 | 10.2 |