# LLM Safety Evaluation on HARMAMP Benign instances Source: [LLM Safety Evaluation on HARMAMP Benign instances]() Collection index: [llms.txt]() Data updated: 2026-06-02T13:27:50.646Z Evaluation of LLM safety performance using the HARMAMP Benign instances dataset, measured by Over-Refusal Rate and Intervention Turn Rate. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | LLM Safety Evaluation | | Dataset (including any stated split) | HARMAMP Benign instances | | Metric | Over-Refusal Rate | | Unit | PERCENT | | Direction | Lower is better | | Results loaded | 30 of 30 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Over-Refusal Rate](): Lower is better; unit: PERCENT. - [Intervention Turn Rate](): Lower is better; unit: PERCENT. ## Recorded results | Method | Over-Refusal Rate | Result date | Report paper | | --- | --- | --- | --- | | Vanilla LLM | 8.9 | Not recorded | 6a1e6426dbd7150506f85805 | | TRAJSAFE | 11.64 | Not recorded | 6a1e6426dbd7150506f85805 | | Vanilla LLM | 12.33 | Not recorded | 6a1e6426dbd7150506f85805 | | SFT Monitor | 15.07 | Not recorded | 6a1e6426dbd7150506f85805 | | TRAJSAFE | 15.07 | Not recorded | 6a1e6426dbd7150506f85805 | | GPT-oss-safeguard | 17.12 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-4 | 17.81 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-3 | 18.49 | Not recorded | 6a1e6426dbd7150506f85805 | | SFT Monitor | 18.49 | Not recorded | 6a1e6426dbd7150506f85805 | | Vanilla LLM | 21.23 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-3 | 21.92 | Not recorded | 6a1e6426dbd7150506f85805 | | TRAJSAFE | 21.92 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-3 | 21.92 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-3 | 23.29 | Not recorded | 6a1e6426dbd7150506f85805 | | SFT Monitor | 25.34 | Not recorded | 6a1e6426dbd7150506f85805 | | Prompting-based Monitor | 26.71 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-4 | 31.51 | Not recorded | 6a1e6426dbd7150506f85805 | | GPT-oss-safeguard | 32.19 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-3 | 32.88 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-3 | 32.88 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-4 | 33.56 | Not recorded | 6a1e6426dbd7150506f85805 | | Prompting-based Monitor | 34.25 | Not recorded | 6a1e6426dbd7150506f85805 | | GPT-oss-safeguard | 34.93 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-4 | 34.93 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-4 | 41.78 | Not recorded | 6a1e6426dbd7150506f85805 | | Prompting-based Monitor | 43.15 | Not recorded | 6a1e6426dbd7150506f85805 | | Llama-Guard-4 | 44.52 | Not recorded | 6a1e6426dbd7150506f85805 | | GPT-oss-safeguard | 53.42 | Not recorded | 6a1e6426dbd7150506f85805 | | GPT-oss-safeguard | 55.48 | Not recorded | 6a1e6426dbd7150506f85805 | | GPT-oss-safeguard | 62.33 | Not recorded | 6a1e6426dbd7150506f85805 | ### Vanilla LLM Over-Refusal Rate: 8.9. Result ID: 6a1ed9e9fbc373592c1aea22. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B"}. This record does not assert independent reproduction. ### TRAJSAFE Over-Refusal Rate: 11.64. Result ID: 6a1ed9e9fbc373592c1aea2b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B"}. This record does not assert independent reproduction. ### Vanilla LLM Over-Refusal Rate: 12.33. Result ID: 6a1ed9e9fbc373592c1aea36. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini"}. This record does not assert independent reproduction. ### SFT Monitor Over-Refusal Rate: 15.07. Result ID: 6a1ed9e9fbc373592c1aea2a. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B"}. This record does not assert independent reproduction. ### TRAJSAFE Over-Refusal Rate: 15.07. Result ID: 6a1ed9e9fbc373592c1aea3f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini"}. This record does not assert independent reproduction. ### GPT-oss-safeguard Over-Refusal Rate: 17.12. Result ID: 6a1ed9e9fbc373592c1aea3c. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### Llama-Guard-4 Over-Refusal Rate: 17.81. Result ID: 6a1ed9e9fbc373592c1aea3a. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### Llama-Guard-3 Over-Refusal Rate: 18.49. Result ID: 6a1ed9e9fbc373592c1aea38. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### SFT Monitor Over-Refusal Rate: 18.49. Result ID: 6a1ed9e9fbc373592c1aea3e. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini"}. This record does not assert independent reproduction. ### Vanilla LLM Over-Refusal Rate: 21.23. Result ID: 6a1ed9e9fbc373592c1aea2c. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B"}. This record does not assert independent reproduction. ### Llama-Guard-3 Over-Refusal Rate: 21.92. Result ID: 6a1ed9e9fbc373592c1aea23. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### TRAJSAFE Over-Refusal Rate: 21.92. Result ID: 6a1ed9e9fbc373592c1aea35. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B"}. This record does not assert independent reproduction. ### Llama-Guard-3 Over-Refusal Rate: 21.92. Result ID: 6a1ed9e9fbc373592c1aea37. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### Llama-Guard-3 Over-Refusal Rate: 23.29. Result ID: 6a1ed9e9fbc373592c1aea24. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### SFT Monitor Over-Refusal Rate: 25.34. Result ID: 6a1ed9e9fbc373592c1aea34. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B"}. This record does not assert independent reproduction. ### Prompting-based Monitor Over-Refusal Rate: 26.71. Result ID: 6a1ed9e9fbc373592c1aea3d. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini"}. This record does not assert independent reproduction. ### Llama-Guard-4 Over-Refusal Rate: 31.51. Result ID: 6a1ed9e9fbc373592c1aea26. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### GPT-oss-safeguard Over-Refusal Rate: 32.19. Result ID: 6a1ed9e9fbc373592c1aea28. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### Llama-Guard-3 Over-Refusal Rate: 32.88. Result ID: 6a1ed9e9fbc373592c1aea2d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### Llama-Guard-3 Over-Refusal Rate: 32.88. Result ID: 6a1ed9e9fbc373592c1aea2e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### Llama-Guard-4 Over-Refusal Rate: 33.56. Result ID: 6a1ed9e9fbc373592c1aea25. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### Prompting-based Monitor Over-Refusal Rate: 34.25. Result ID: 6a1ed9e9fbc373592c1aea29. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B"}. This record does not assert independent reproduction. ### GPT-oss-safeguard Over-Refusal Rate: 34.93. Result ID: 6a1ed9e9fbc373592c1aea32. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### Llama-Guard-4 Over-Refusal Rate: 34.93. Result ID: 6a1ed9e9fbc373592c1aea39. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### Llama-Guard-4 Over-Refusal Rate: 41.78. Result ID: 6a1ed9e9fbc373592c1aea30. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B","Protocol":"prompt-response"}. This record does not assert independent reproduction. ### Prompting-based Monitor Over-Refusal Rate: 43.15. Result ID: 6a1ed9e9fbc373592c1aea33. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B"}. This record does not assert independent reproduction. ### Llama-Guard-4 Over-Refusal Rate: 44.52. Result ID: 6a1ed9e9fbc373592c1aea2f. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### GPT-oss-safeguard Over-Refusal Rate: 53.42. Result ID: 6a1ed9e9fbc373592c1aea3b. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"GPT-5-mini","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### GPT-oss-safeguard Over-Refusal Rate: 55.48. Result ID: 6a1ed9e9fbc373592c1aea27. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Llama-3.1-8B","Protocol":"prompt-only"}. This record does not assert independent reproduction. ### GPT-oss-safeguard Over-Refusal Rate: 62.33. Result ID: 6a1ed9e9fbc373592c1aea31. [Reporting paper]() Recorded evaluation configuration: {"Target Model":"Qwen3-4B","Protocol":"prompt-only"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.