Safety Evaluation on AdvBench (Adversarial Attack Metrics)
100Overall Safety ScorePost-hoc (LlamaGuard)
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Post-hoc (LlamaGuard)Base Model=Llama3.1-8B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 37.5 | 46.15 | 57.5 | 98.65 | 85.77 | 60.96 | 92.31 | 72.36 | — | — | |
| STAIR-DPOBase Model=Llama3.1-8B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 53.27 | 28.65 | 94.81 | 100 | 69.04 | 65 | 97.88 | 76.08 | — | — | |
| WJ-SFTBase Model=Llama3.1-8B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 32.88 | 27.69 | 68.65 | 100 | 61.15 | 59.23 | 99.62 | 68.65 | — | — | |
| ReSA-SFTBase Model=Llama3.1-8B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 87.12 | 64.23 | 97.31 | 100 | 81.73 | 88.65 | 99.62 | 89.83 | — | — | |
| STAIR-DPOBase Model=Llama3.1-8B-Instruct, Evaluator=HarmBench Classifier2025.09 | 100 | 90.38 | 85.19 | 99.62 | 100 | 92.12 | 85.19 | 99.42 | 93.99 | — | — | |
| WJ-SFTBase Model=Llama3.1-8B-Instruct, Evaluator=HarmBench Classifier2025.09 | 100 | 75 | 75.96 | 84.62 | 100 | 86.54 | 70.77 | 99.62 | 86.56 | — | — | |
| ReSA-SFTBase Model=Llama3.1-8B-Instruct, Evaluator=HarmBench Classifier2025.09 | 100 | 97.5 | 94.23 | 99.42 | 100 | 98.85 | 95.19 | 99.62 | 98.1 | — | — | |
| BaseBase Model=Qwen2.5-7B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 16.35 | 6.92 | 31.54 | 43.65 | 5.58 | 14.23 | 17.88 | 29.52 | — | — | |
| Post-hoc (LlamaGuard)Base Model=Qwen2.5-7B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 29.81 | 51.35 | 49.23 | 98.85 | 87.31 | 63.08 | 92.12 | 71.47 | — | — | |
| STAIR-DPO*Base Model=Qwen2.5-7B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 54.62 | 28.65 | 94.62 | 84.42 | 34.04 | 62.88 | 94.81 | 69.25 | — | — | |
| WJ-SFTBase Model=Qwen2.5-7B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 21.15 | 20.38 | 64.23 | 98.85 | 34.81 | 42.88 | 97.88 | 60.02 | — | — | |
| ReSA-SFTBase Model=Qwen2.5-7B-Instruct, Evaluator=LlamaGuard2025.09 | 100 | 85.96 | 64.23 | 96.73 | 98.08 | 74.62 | 89.04 | 98.08 | 88.34 | — | — | |
| STAIR-DPO*Base Model=Qwen2.5-7B-Instruct, Evaluator=HarmBench Classifier2025.09 | 100 | 90.96 | 87.69 | 99.62 | 96.15 | 65.58 | 81.73 | 99.04 | 90.1 | — | — | |
| ReSA-SFTBase Model=Qwen2.5-7B-Instruct, Evaluator=HarmBench Classifier2025.09 | 100 | 96.35 | 94.04 | 98.85 | 100 | 98.27 | 94.23 | 99.42 | 97.65 | — | — | |
| STAIR-DPOBase Model=Llama3.1-8B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 99.92 | 74.24 | 63.83 | 95.92 | 99.92 | 84.41 | 74.95 | 99.02 | 86.53 | — | — | |
| STAIR-DPO*Base Model=Qwen2.5-7B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 99.9 | 75.61 | 62.43 | 95.91 | 95.7 | 65.36 | 70.65 | 98.13 | 82.96 | — | — | |
| WJ-SFTBase Model=Qwen2.5-7B-Instruct, Evaluator=HarmBench Classifier2025.09 | 99.81 | 68.08 | 63.65 | 81.35 | 99.04 | 65.96 | 59.23 | 98.65 | 79.47 | — | — | |
| ReSA-SFTBase Model=Qwen2.5-7B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 99.68 | 88.57 | 73.38 | 97.46 | 99.6 | 92.58 | 90.06 | 97.99 | 92.42 | — | — | |
| WJ-SFTBase Model=Qwen2.5-7B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 99.58 | 47.96 | 48.19 | 70.83 | 99.19 | 65.22 | 53.92 | 97.27 | 72.77 | — | — | |
| WJ-SFTBase Model=Llama3.1-8B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 99.57 | 56.06 | 56.93 | 73.29 | 99.83 | 86.21 | 65.89 | 98.97 | 79.59 | — | — | |
| ReSA-SFTBase Model=Llama3.1-8B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 99.39 | 89.85 | 74.1 | 97.44 | 99.65 | 94.93 | 89.72 | 99.1 | 93.02 | — | — | |
| BaseBase Model=Qwen2.5-7B-Instruct, Evaluator=HarmBench Classifier2025.09 | 99.23 | 63.85 | 40.38 | 70.58 | 48.65 | 31.73 | 23.65 | 67.31 | 55.67 | — | — | |
| Post-hoc (LlamaGuard)Base Model=Qwen2.5-7B-Instruct, Evaluator=HarmBench Classifier2025.09 | 99.23 | 75.77 | 76.54 | 82.69 | 96.73 | 97.12 | 68.85 | 98.65 | 86.95 | — | — | |
| Post-hoc (LlamaGuard)Base Model=Llama3.1-8B-Instruct, Evaluator=HarmBench Classifier2025.09 | 98.46 | 76.54 | 75.38 | 79.81 | 97.5 | 96.73 | 71.54 | 97.69 | 86.71 | — | — | |
| Post-hoc (LlamaGuard)Base Model=Llama3.1-8B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 98.16 | 52.83 | 63.75 | 61.23 | 97.54 | 92.72 | 69.34 | 87.93 | 77.94 | — | — | |
| BaseBase Model=Qwen2.5-7B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 95.85 | 36.08 | 30.06 | 33.52 | 57.47 | 34.45 | 28.58 | 36.79 | 44.1 | — | — | |
| Post-hoc (LlamaGuard)Base Model=Qwen2.5-7B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 95.85 | 46.78 | 64.99 | 48.6 | 97.08 | 94.86 | 67.03 | 90.2 | 75.67 | — | — | |
| BaseBase Model=Llama3.1-8B-Instruct, Evaluator=LlamaGuard2025.09 | 95.38 | 21.35 | 20.38 | 46.92 | 42.69 | 46.92 | 39.62 | 27.31 | 42.57 | — | — | |
| BaseBase Model=Llama3.1-8B-Instruct, Evaluator=Fine-tuned StrongREJECT Evaluator2025.09 | 94.71 | 39.57 | 42.52 | 52.34 | 49.55 | 63.59 | 51.94 | 38.07 | 54.04 | — | — | |
| BaseBase Model=Llama3.1-8B-Instruct, Evaluator=HarmBench Classifier2025.09 | 94.23 | 63.65 | 54.42 | 70.96 | 44.62 | 67.12 | 50.96 | 72.12 | 64.76 | — | — | |
| GuardReasonerBackbone=R1-Qwen-7B, Defense Strategy=Post-hoc Guard Model2025.08 | — | — | — | — | — | — | — | — | — | 2 | 6 | |
| GuardReasonerBackbone=R1-Llama-8B, Defense Strategy=Post-hoc Guard Model2025.08 | — | — | — | — | — | — | — | — | — | 0 | 0 | |
| Llama Guard 3Backbone=R1-Qwen-7B, Defense Strategy=Post-hoc Guard Model2025.08 | — | — | — | — | — | — | — | — | — | 0 | 10 | |
| Llama Guard 3Backbone=R1-Llama-8B, Defense Strategy=Post-hoc Guard Model2025.08 | — | — | — | — | — | — | — | — | — | 2 | 2 | |
| No DefenseBackbone=R1-Qwen-7B, Defense Strategy=None2025.08 | — | — | — | — | — | — | — | — | — | 40 | 32 | |
| No DefenseBackbone=R1-Llama-8B, Defense Strategy=None2025.08 | — | — | — | — | — | — | — | — | — | 26 | 22 | |
| PPLBackbone=R1-Qwen-7B, Defense Strategy=Input-level Filtering2025.08 | — | — | — | — | — | — | — | — | — | 40 | 32 | |
| PPLBackbone=R1-Llama-8B, Defense Strategy=Input-level Filtering2025.08 | — | — | — | — | — | — | — | — | — | 26 | 22 | |
| ReasoningGuardBackbone=R1-Qwen-7B, Defense Strategy=Proactive Reasoning Guard2025.08 | — | — | — | — | — | — | — | — | — | 2 | 2 | |
| ReasoningGuardBackbone=R1-Llama-8B, Defense Strategy=Proactive Reasoning Guard2025.08 | — | — | — | — | — | — | — | — | — | 0 | 0 | |
| ReasoningGuard + GRBackbone=R1-Qwen-7B, Defense Strategy=Hybrid2025.08 | — | — | — | — | — | — | — | — | — | 0 | 0 | |
| ReasoningGuard + GRBackbone=R1-Llama-8B, Defense Strategy=Hybrid2025.08 | — | — | — | — | — | — | — | — | — | 0 | 0 | |
| ReasoningGuard + LGBackbone=R1-Qwen-7B, Defense Strategy=Hybrid2025.08 | — | — | — | — | — | — | — | — | — | 0 | 0 | |
| ReasoningGuard + LGBackbone=R1-Llama-8B, Defense Strategy=Hybrid2025.08 | — | — | — | — | — | — | — | — | — | 0 | 0 |