Safety Overrefusal Evaluation on Overrefusal Evaluation Suite (test)
1XSTestGemma2-9B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Gemma2-9BDefense Method=None2026.02 | 1 | 0.974 | 0.987 | |
| Augmented RTBase Model=Gemma2-9B2026.02 | 1 | 0.974 | 0.987 | |
| Llama3-8BDefense Method=None2026.02 | 0.98 | 0.989 | 0.985 | |
| Gemma2-2BDefense Method=None2026.02 | 0.98 | 0.975 | 0.978 | |
| Fail-Closed AlignmentBase Model=Llama3-8B2026.02 | 0.92 | 0.983 | 0.952 | |
| Fail-Closed AlignmentBase Model=Gemma2-2B2026.02 | 0.88 | 0.971 | 0.926 | |
| LATBase Model=Gemma2-2B2026.02 | 0.76 | 0.944 | 0.852 | |
| Fail-Closed AlignmentBase Model=Gemma2-9B2026.02 | 0.76 | 0.962 | 0.861 | |
| Llama2-7BDefense Method=None2026.02 | 0.75 | 0.993 | 0.872 | |
| LATBase Model=Gemma2-9B2026.02 | 0.74 | 0.937 | 0.839 | |
| CATBase Model=Gemma2-9B2026.02 | 0.72 | 0.986 | 0.853 | |
| DeepRefusalBase Model=Llama3-8B2026.02 | 0.7 | 0.965 | 0.833 | |
| CATBase Model=Gemma2-2B2026.02 | 0.64 | 0.969 | 0.805 | |
| DeepRefusalBase Model=Gemma2-9B2026.02 | 0.62 | 0.965 | 0.793 | |
| DeepRefusalBase Model=Llama2-7B2026.02 | 0.6 | 0.958 | 0.779 | |
| LATBase Model=Llama3-8B2026.02 | 0.6 | 0.978 | 0.789 | |
| Fail-Closed AlignmentBase Model=Llama2-7B2026.02 | 0.46 | 0.974 | 0.717 | |
| Augmented RTBase Model=Gemma2-2B2026.02 | 0.4 | 0.974 | 0.687 | |
| DeepRefusalBase Model=Gemma2-2B2026.02 | 0.34 | 0.901 | 0.621 | |
| Augmented RTBase Model=Llama2-7B2026.02 | 0.3 | 0.966 | 0.633 | |
| Augmented RTBase Model=Llama3-8B2026.02 | 0.26 | 0.968 | 0.614 | |
| CATBase Model=Llama2-7B2026.02 | 0.2 | 0.959 | 0.58 | |
| CATBase Model=Llama3-8B2026.02 | 0.18 | 0.85 | 0.515 | |
| LATBase Model=Llama2-7B2026.02 | 0.1 | 0.868 | 0.484 |