Over Refusal on Over Refusal scenario
100ASRHQQ
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| HQQModel=Qwen2.5 7B, Type=DIOB2026.05 | 100 | — | — | |
| SINQModel=Qwen2.5 7B, Type=DIOB2026.05 | 99.9 | — | — | |
| AWQModel=Qwen2.5 7B, Type=DDOB2026.05 | 99.5 | — | — | |
| GGUF Q4_0Model=Qwen2.5 7B, Type=Zero-shot2026.05 | 98.3 | — | — | |
| NF4Model=Qwen2.5 7B, Type=Zero-shot2026.05 | 95.8 | — | — | |
| LLM.int8()Model=Qwen2.5 7B, Type=Zero-shot2026.05 | 95 | — | — | |
| GPTQ 4-bitModel=Qwen2.5 7B, Type=DDOB2026.05 | 94.3 | — | — | |
| FP4Model=Qwen2.5 7B, Type=Zero-shot2026.05 | 93.4 | — | — | |
| GGUF Q4_K_MModel=Qwen2.5 7B, Type=DIOB2026.05 | 92.9 | — | — | |
| HQQModel=Llama3.1 8B, Type=DIOB2026.05 | 92.9 | — | — | |
| SINQModel=Llama3.1 8B, Type=DIOB2026.05 | 89.6 | — | — | |
| LLM.int8()Model=Llama3.1 8B, Type=Zero-shot2026.05 | 89.1 | — | — | |
| GGUF IQ4_XSModel=Llama3.1 8B, Type=DDOB2026.05 | 85.1 | — | — | |
| FP4Model=Llama3.1 8B, Type=Zero-shot2026.05 | 84.4 | — | — | |
| GGUF Q4_K_MModel=Llama3.1 8B, Type=DIOB2026.05 | 81.9 | — | — | |
| GPTQ 8-bitModel=Qwen2.5 7B, Type=DDOB2026.05 | 78.3 | — | — | |
| AWQModel=Llama3.1 8B, Type=DDOB2026.05 | 74.3 | — | — | |
| NF4Model=Llama3.1 8B, Type=Zero-shot2026.05 | 73.7 | — | — | |
| GGUF Q4_0Model=Llama3.1 8B, Type=Zero-shot2026.05 | 70.4 | — | — | |
| GPTQ 4-bitModel=Llama3.1 8B, Type=DDOB2026.05 | 68.8 | — | — | |
| GPTQ 8-bitModel=Llama3.1 8B, Type=DDOB2026.05 | 66.9 | — | — | |
| AWQModel=Mistral 7B, Type=DDOB2026.05 | 49.6 | — | — | |
| GPTQ 4-bitModel=Mistral 7B, Type=DDOB2026.05 | 48.7 | — | — | |
| LLM.int8()Model=Mistral 7B, Type=Zero-shot2026.05 | 48.3 | — | — | |
| NF4Model=Mistral 7B, Type=Zero-shot2026.05 | 47.7 | — | — | |
| SINQModel=Mistral 7B, Type=DIOB2026.05 | 43 | — | — | |
| AutoRoundModel=Mistral 7B, Type=DDOB2026.05 | 41.7 | — | — | |
| AutoRoundModel=Qwen2.5 7B, Type=DDOB2026.05 | 37.9 | — | — | |
| HQQModel=Mistral 7B, Type=DIOB2026.05 | 33.7 | — | — | |
| FP4Model=Mistral 7B, Type=Zero-shot2026.05 | 27.7 | — | — | |
| GPTQ 8-bitModel=Mistral 7B, Type=DDOB2026.05 | 25.1 | — | — | |
| GGUF Q4_K_MModel=Mistral 7B, Type=DIOB2026.05 | 13.5 | — | — | |
| AutoRoundModel=Llama3.1 8B, Type=DDOB2026.05 | 10.6 | — | — | |
| GGUF IQ4_XSModel=Qwen2.5 7B, Type=DDOB2026.05 | 7.7 | — | — | |
| GGUF IQ4_XSModel=Mistral 7B, Type=DDOB2026.05 | 5.3 | — | — | |
| AttackedModel=Mistral 7B, Precision=BF162026.05 | 4.3 | — | — | |
| GGUF Q4_0Model=Mistral 7B, Type=Zero-shot2026.05 | 4.3 | — | — | |
| AttackedModel=Llama3.1 8B, Precision=BF162026.05 | 1.2 | — | — | |
| OriginalModel=Qwen2.5 7B, Precision=BF162026.05 | 0.8 | — | — | |
| OriginalModel=Mistral 7B, Precision=BF162026.05 | 0.8 | — | — | |
| AttackedModel=Qwen2.5 7B, Precision=BF162026.05 | 0.3 | — | — | |
| OriginalModel=Llama3.1 8B, Precision=BF162026.05 | 0.2 | — | — | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=Unpruned, Sparsity Ratio=-2025.10 | — | 0.5 | 0.5 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=Mag., Sparsity Ratio=20%2025.10 | — | 95.5 | 0.3 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=SparseGPT, Sparsity Ratio=20%2025.10 | — | 70.4 | 0.3 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=SparseGPT, Sparsity Ratio=2:42025.10 | — | 21.4 | 2.9 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=SparseGPT, Sparsity Ratio=50%2025.10 | — | 78.3 | 2.7 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=Wanda, Sparsity Ratio=20%2025.10 | — | 93 | 0.8 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=Wanda, Sparsity Ratio=2:42025.10 | — | 63.2 | 1.8 | |
| Llama3.1-8BModel=Llama3.1-8B, Pruning Method=Wanda, Sparsity Ratio=50%2025.10 | — | 97.3 | 2.9 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=Unpruned, Sparsity Ratio=-2025.10 | — | 2.1 | 2.5 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=Mag., Sparsity Ratio=20%2025.10 | — | 92.7 | 2.1 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=SparseGPT, Sparsity Ratio=20%2025.10 | — | 78.8 | 2.6 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=SparseGPT, Sparsity Ratio=2:42025.10 | — | 47.9 | 8.7 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=SparseGPT, Sparsity Ratio=50%2025.10 | — | 98.7 | 4.6 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=Wanda, Sparsity Ratio=20%2025.10 | — | 91.1 | 2.1 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=Wanda, Sparsity Ratio=2:42025.10 | — | 78.7 | 6.9 | |
| OLMo-2-7BModel=OLMo-2-7B, Pruning Method=Wanda, Sparsity Ratio=50%2025.10 | — | 97.2 | 4.1 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=Unpruned, Sparsity Ratio=-2025.10 | — | 1.1 | 0.4 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=Mag., Sparsity Ratio=20%2025.10 | — | 93.9 | 0.3 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=SparseGPT, Sparsity Ratio=20%2025.10 | — | 51.3 | 0.9 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=SparseGPT, Sparsity Ratio=2:42025.10 | — | 40.9 | 2.8 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=SparseGPT, Sparsity Ratio=50%2025.10 | — | 67.8 | 1.1 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=Wanda, Sparsity Ratio=20%2025.10 | — | 93.7 | 0.8 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=Wanda, Sparsity Ratio=2:42025.10 | — | 96.3 | 4.1 | |
| Qwen2.5-7BModel=Qwen2.5-7B, Pruning Method=Wanda, Sparsity Ratio=50%2025.10 | — | 98.4 | 1.7 |