LLM Safety Evaluation on HARMAMP Benign instances
8.9Over-Refusal RateVanilla LLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Vanilla LLMTarget Model=Llama-3.1-8B2026.06 | 8.9 | — | |
| TRAJSAFETarget Model=Llama-3.1-8B2026.06 | 11.64 | 5.41 | |
| Vanilla LLMTarget Model=GPT-5-mini2026.06 | 12.33 | — | |
| SFT MonitorTarget Model=Llama-3.1-8B2026.06 | 15.07 | 22.62 | |
| TRAJSAFETarget Model=GPT-5-mini2026.06 | 15.07 | 5 | |
| GPT-oss-safeguardTarget Model=GPT-5-mini, Protocol=prompt-response2026.06 | 17.12 | 4.21 | |
| Llama-Guard-4Target Model=GPT-5-mini, Protocol=prompt-response2026.06 | 17.81 | 2.41 | |
| Llama-Guard-3Target Model=GPT-5-mini, Protocol=prompt-response2026.06 | 18.49 | 4.63 | |
| SFT MonitorTarget Model=GPT-5-mini2026.06 | 18.49 | 19.45 | |
| Vanilla LLMTarget Model=Qwen3-4B2026.06 | 21.23 | — | |
| Llama-Guard-3Target Model=Llama-3.1-8B, Protocol=prompt-only2026.06 | 21.92 | 7.26 | |
| TRAJSAFETarget Model=Qwen3-4B2026.06 | 21.92 | 5.83 | |
| Llama-Guard-3Target Model=GPT-5-mini, Protocol=prompt-only2026.06 | 21.92 | 5.48 | |
| Llama-Guard-3Target Model=Llama-3.1-8B, Protocol=prompt-response2026.06 | 23.29 | 9.23 | |
| SFT MonitorTarget Model=Qwen3-4B2026.06 | 25.34 | 21.79 | |
| Prompting-based MonitorTarget Model=GPT-5-mini2026.06 | 26.71 | 96.31 | |
| Llama-Guard-4Target Model=Llama-3.1-8B, Protocol=prompt-response2026.06 | 31.51 | 11.59 | |
| GPT-oss-safeguardTarget Model=Llama-3.1-8B, Protocol=prompt-response2026.06 | 32.19 | 13 | |
| Llama-Guard-3Target Model=Qwen3-4B, Protocol=prompt-only2026.06 | 32.88 | 6.18 | |
| Llama-Guard-3Target Model=Qwen3-4B, Protocol=prompt-response2026.06 | 32.88 | 6.22 | |
| Llama-Guard-4Target Model=Llama-3.1-8B, Protocol=prompt-only2026.06 | 33.56 | 10.07 | |
| Prompting-based MonitorTarget Model=Llama-3.1-8B2026.06 | 34.25 | 94.25 | |
| GPT-oss-safeguardTarget Model=Qwen3-4B, Protocol=prompt-response2026.06 | 34.93 | 7.78 | |
| Llama-Guard-4Target Model=GPT-5-mini, Protocol=prompt-only2026.06 | 34.93 | 11.21 | |
| Llama-Guard-4Target Model=Qwen3-4B, Protocol=prompt-response2026.06 | 41.78 | 10.59 | |
| Prompting-based MonitorTarget Model=Qwen3-4B2026.06 | 43.15 | 95.68 | |
| Llama-Guard-4Target Model=Qwen3-4B, Protocol=prompt-only2026.06 | 44.52 | 10.36 | |
| GPT-oss-safeguardTarget Model=GPT-5-mini, Protocol=prompt-only2026.06 | 53.42 | 19.59 | |
| GPT-oss-safeguardTarget Model=Llama-3.1-8B, Protocol=prompt-only2026.06 | 55.48 | 23.24 | |
| GPT-oss-safeguardTarget Model=Qwen3-4B, Protocol=prompt-only2026.06 | 62.33 | 23.12 |