Safety Evaluation on DirectHarm 4
9Attack Success RateGradSafe
Evaluation results
| Method | Links | |
|---|---|---|
| GradSafeModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 9 | |
| InstructModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 9.5 | |
| InstructModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 9.5 | |
| Base ModelBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=None, Filtering Method=None2026.05 | 9.5 | |
| InstructModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 11 | |
| InstructModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 11 | |
| Base ModelBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=None, Filtering Method=None2026.05 | 11 | |
| InstructModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 12.75 | |
| InstructModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 12.75 | |
| Base ModelBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=None, Filtering Method=None2026.05 | 12.75 | |
| DataShieldBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Ours2026.05 | 14.25 | |
| LARFBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=LARF2026.05 | 15 | |
| SEALBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=SEAL2026.05 | 16.25 | |
| GradSafeBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=GradSafe2026.05 | 17.75 | |
| Bi-AnchoringBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Bi-Anchoring2026.05 | 17.75 | |
| SEALModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 20 | |
| RandomModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 21.5 | |
| Without FilteringBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Without Filtering2026.05 | 23.75 | |
| RandomModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 26.25 | |
| RandomModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 26.25 | |
| SEALModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 26.75 | |
| RandomModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 26.75 | |
| DataShieldBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Ours2026.05 | 26.75 | |
| SEALModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 27.75 | |
| GradSafeModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 27.75 | |
| SEALModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 28.25 | |
| Bi-AnchorModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 28.5 | |
| GradSafeModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 30.5 | |
| DataShieldBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Ours2026.05 | 32.5 | |
| LARFBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=LARF2026.05 | 34.75 | |
| RandomModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 36.75 | |
| RandomModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 37 | |
| Bi-AnchorModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 37.25 | |
| LARFModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 38.25 | |
| LARFBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=LARF2026.05 | 39.5 | |
| GuardModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 40.25 | |
| GuardModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 40.75 | |
| SEALBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=SEAL2026.05 | 42.5 | |
| GuardModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 43.75 | |
| SEALBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=SEAL2026.05 | 43.75 | |
| GuardModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 44.25 | |
| LARFModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 45.25 | |
| DataShieldModel=Qwen2.5, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 45.5 | |
| DataShieldModel=Llama3.1, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 47 | |
| DataShieldBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Ours2026.05 | 47 | |
| SEALModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 49.75 | |
| LARFBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=LARF2026.05 | 50.25 | |
| LARFModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 50.5 | |
| Bi-AnchorModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 51.25 | |
| DataShieldBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Ours2026.05 | 51.25 | |
| Bi-AnchoringBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Bi-Anchoring2026.05 | 52 | |
| LARFBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=LARF2026.05 | 52.25 | |
| GradSafeModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 53.75 | |
| SEALBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=SEAL2026.05 | 53.75 | |
| SEALBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=SEAL2026.05 | 55.25 | |
| Bi-AnchoringBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Bi-Anchoring2026.05 | 56.5 | |
| GradSafeBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=GradSafe2026.05 | 58.25 | |
| GradSafeBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=GradSafe2026.05 | 59.75 | |
| DataShieldBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Ours2026.05 | 60 | |
| Bi-AnchoringBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Bi-Anchoring2026.05 | 60.25 | |
| Without FilteringBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Without Filtering2026.05 | 61.75 | |
| Without FilteringBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Alpaca Dataset, Filtering Method=Without Filtering2026.05 | 61.75 | |
| LARFBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=LARF2026.05 | 62.5 | |
| DataShieldModel=Llama3, Fine-tuning Source=Alpaca, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 63.75 | |
| GradSafeModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 66 | |
| SEALBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=SEAL2026.05 | 66.75 | |
| Bi-AnchoringBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Bi-Anchoring2026.05 | 71.5 | |
| SEALModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 71.75 | |
| Bi-AnchorModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 72.5 | |
| GradSafeBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=GradSafe2026.05 | 74.25 | |
| GradSafeBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=GradSafe2026.05 | 74.25 | |
| Bi-AnchorModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 75.75 | |
| LARFModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 76 | |
| GradSafeModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 77.5 | |
| Bi-AnchoringBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Bi-Anchoring2026.05 | 78.5 | |
| Without FilteringBackbone=Qwen2.5-7B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Without Filtering2026.05 | 78.5 | |
| GradSafeBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=GradSafe2026.05 | 78.75 | |
| Without FilteringBackbone=Llama3-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Without Filtering2026.05 | 79 | |
| Bi-AnchorModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 79.25 | |
| GuardModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 80 | |
| Standard SFTBackbone=Llama3.1-8B-Instruct, Fine-tuning Dataset=Dolly Dataset, Filtering Method=Standard SFT2026.05 | 80.5 | |
| LARFModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 81.5 | |
| GuardModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 82.5 | |
| LARFModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 82.75 | |
| DataShieldModel=Qwen2.5, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 83.5 | |
| DataShieldModel=Llama3.1, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 84.75 | |
| DataShieldModel=Llama3, Fine-tuning Source=Dolly, Selection Protocol=Top-1000 safety-degrading samples2026.05 | 91 |