Prefix-based Violation Detection on POLICYGUARDBENCH
93.89Accuracy (N=1)LLAMA-4-SCOUT-17B-INSTRUCT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| LLAMA-4-SCOUT-17B-INSTRUCTModel family=Llama family, Model variant=17B-INSTRUCT2025.10 | 93.89 | 88.54 | 85.83 | 83.55 | 82.37 | 86.84 | |
| LLAMA-3.3-70B-INSTRUCTModel family=Llama family, Model variant=70B-INSTRUCT2025.10 | 92.98 | 84.41 | 83.68 | 83.05 | 81.91 | 85.21 | |
| DEEPSEEK-V3.1Model family=Frontier Models, Model mode=NON-THINKING MODE2025.10 | 92.71 | 85.87 | 84.67 | 83.04 | 81.29 | 85.52 | |
| GEMMA-3-12B-ITModel family=Gemma family, Model variant=12B-IT2025.10 | 92.27 | 84.92 | 83.64 | 82.58 | 82.03 | 85.09 | |
| GEMMA-3-27B-ITModel family=Gemma family, Model variant=27B-IT2025.10 | 91.08 | 87.89 | 85.26 | 82.54 | 80.99 | 85.55 | |
| POLICYGUARD-4BModel family=Ours, Model parameters=4B2025.10 | 91.01 | 86.48 | 84.41 | 82.76 | 81.9 | 85.31 | |
| LLAMA-3.2-3B-INSTRUCTModel family=Llama family, Model variant=3B-INSTRUCT2025.10 | 90.86 | 81.99 | 73.48 | 63.77 | 56.93 | 73.41 | |
| CLAUDE-SONNET-4Model family=Frontier Models2025.10 | 89.94 | 85.37 | 84.84 | 83.09 | 81.15 | 84.88 | |
| GEMINI-1.5-PROModel family=Frontier Models2025.10 | 89.9 | 87.79 | 86.67 | 86.3 | 85.43 | 87.22 | |
| LLAMA-3.1-8B-INSTRUCTModel family=Llama family, Model variant=8B-INSTRUCT2025.10 | 89.76 | 77.41 | 67.31 | 61.21 | 53.71 | 69.88 | |
| QWEN3-235B-A22B-INSTRUCTModel family=Qwen family, Model variant=235B-A22B-INSTRUCT2025.10 | 89.76 | 87.52 | 86.44 | 85.69 | 84.98 | 86.88 | |
| QWEN3-4B-INSTRUCTModel family=Qwen family, Model variant=4B-INSTRUCT2025.10 | 88.32 | 82.31 | 80.38 | 76.88 | 73.3 | 80.24 | |
| QWEN2.5-72B-INSTRUCTModel family=Qwen family, Model variant=72B-INSTRUCT2025.10 | 85.08 | 81.7 | 83.82 | 84.04 | 83.27 | 83.58 | |
| GEMMA-3-4B-ITModel family=Gemma family, Model variant=4B-IT2025.10 | 82.6 | 64.01 | 57.81 | 62.36 | 63.85 | 66.13 | |
| QWEN3-30B-A3B-INSTRUCTModel family=Qwen family, Model variant=30B-A3B-INSTRUCT2025.10 | 71.99 | 69.55 | 72.13 | 72.73 | 74.68 | 72.22 | |
| QWEN3-8BModel family=Qwen family, Model variant=8B2025.10 | 61.02 | 74.29 | 63.08 | 55.26 | 49.11 | 60.55 |