Violation Detection on PolicyGuardBench
88.83Safety F1LLAMA-3.3-70B-INSTRUCT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LLAMA-3.3-70B-INSTRUCTType=IT, Size=70B2025.10 | 88.83 | 90.54 | 305 | |
| GPT-4oFamily=Instruction-following, In-distribution status=false2026.05 | 87.77 | — | — | |
| GEMMA-3-12BType=IT, Size=12B2025.10 | 87.73 | 89.64 | 51.3 | |
| POLICYGUARD-4BType=FT, Size=4B2025.10 | 87.59 | 90.14 | 22.5 | |
| QWEN3-235B-A22B-INSTRUCTType=IT, Size=235B2025.10 | 86.9 | 88.69 | 3,640 | |
| CLAUDE-SONNET-4Type=CLOSED, Size=–2025.10 | 86.78 | 89.83 | 1,238 | |
| QWEN2.5-72B-INSTRUCTType=IT, Size=72B2025.10 | 86.07 | 88.25 | 205 | |
| GEMMA-3-27BType=IT, Size=27B2025.10 | 85.2 | 88.5 | 73.6 | |
| GEMINI-1.5-PROType=CLOSED, Size=–2025.10 | 85.02 | 87.13 | 596.1 | |
| DEEPSEEK-V3.1 (NON-THINKING)Type=OPEN, Size=685B2025.10 | 84.07 | 86.13 | 3,270 | |
| LLAMA-4-SCOUT-17B-INSTRUCTType=IT, Size=109B2025.10 | 81.98 | 84.57 | 265 | |
| LPG-4B (Ours)Family=Dynamic guardrail, In-distribution status=false2026.05 | 77.85 | — | — | |
| GuardReasoner-8BFamily=Static guardrail, In-distribution status=false2026.05 | 77.64 | — | — | |
| GEMMA-3-4BType=IT, Size=4B2025.10 | 77.64 | 78.76 | 70.8 | |
| DynaGuard-4BFamily=Dynamic guardrail, In-distribution status=false2026.05 | 77.02 | — | — | |
| QWEN3-30B-A3B-INSTRUCTType=IT, Size=31B2025.10 | 67.2 | 61.83 | 250 | |
| QWEN3-8BType=PT, Size=8B2025.10 | 64.07 | 64.08 | 115.8 | |
| LLAMA GUARDType=GUARDRAIL, Size=7B2025.10 | 59.57 | 42.56 | 87.5 | |
| LLAMA GUARD-4Type=GUARDRAIL, Size=12B2025.10 | 59.54 | 42.39 | 175.3 | |
| LlamaGuard3-8BFamily=Static guardrail, In-distribution status=false2026.05 | 59.52 | — | — | |
| LLAMA GUARD-3Type=GUARDRAIL, Size=8B2025.10 | 59.52 | 42.46 | 164.8 | |
| Qwen3-4BFamily=Instruction-following, In-distribution status=false2026.05 | 53.48 | — | — | |
| QWEN3-4B-INSTRUCTType=IT, Size=4B2025.10 | 53.48 | 68.97 | 25.6 | |
| LLAMA-3.1-8B-INSTRUCTType=IT, Size=8B2025.10 | 42.22 | 66.47 | 85 | |
| ShieldGemma-9BFamily=Static guardrail, In-distribution status=false2026.05 | 34.72 | — | — | |
| SHIELDGEMMA-9BType=GUARDRAIL, Size=9B2025.10 | 34.72 | 54.57 | 39.8 | |
| SHIELDGEMMA-2BType=GUARDRAIL, Size=2B2025.10 | 33.17 | 57.35 | 32.6 | |
| LLAMA-3.2-3B-INSTRUCTType=IT, Size=3B2025.10 | 27.67 | 60.67 | 44.3 | |
| SHIELDGEMMA-27BType=GUARDRAIL, Size=27B2025.10 | 18.34 | 55.55 | 45 | |
| LLAMA GUARD-2Type=GUARDRAIL, Size=8B2025.10 | 0.16 | 57.53 | 40 |