Safety Alignment on HH-RLHF
1.09MD RateShaPO-T
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ShaPO-TBackbone=Qwen2.5-7B2026.02 | 1.09 | 0.12 | |
| ShaPO-RBackbone=Qwen2.5-7B2026.02 | 1.31 | 0.12 | |
| wDPOBackbone=Qwen-2.5-7B2026.03 | 1.64 | 0.15 | |
| ShaPO-TBackbone=LLaMA-3.2-3B2026.02 | 1.75 | 0.23 | |
| ShaPO-RBackbone=LLaMA-3-8B2026.02 | 1.86 | 0.12 | |
| ShaPO-RBackbone=LLaMA-3.2-3B2026.02 | 2.09 | 0.33 | |
| Dr.DPOBackbone=Qwen2.5-7B2026.02 | 2.39 | 0.22 | |
| Dr.DPOBackbone=Qwen-2.5-7B2026.03 | 2.39 | 0.22 | |
| Llama-3.2-3B + wDPOBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 2.39 | 0.33 | |
| Dr.DPOBackbone=LLaMA-3.2-3B2026.02 | 2.57 | 0.34 | |
| Llama-3.2-3B + Dr.DPOBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 2.57 | 0.34 | |
| ShaPO-TBackbone=LLaMA-3-8B2026.02 | 2.94 | 0.19 | |
| wDPOBackbone=Llama-3-8B2026.03 | 4.17 | 0.39 | |
| Dr.DPOBackbone=LLaMA-3-8B2026.02 | 4.58 | 0.37 | |
| Dr.DPOBackbone=Llama-3-8B2026.03 | 4.58 | 0.37 | |
| rDPOBackbone=LLaMA-3.2-3B2026.02 | 5.92 | 2.02 | |
| Llama-3.2-3B + rDPOBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 5.92 | 2.02 | |
| rDPOBackbone=Qwen2.5-7B2026.02 | 6.14 | 1.76 | |
| rDPOBackbone=Qwen-2.5-7B2026.03 | 6.14 | 1.76 | |
| rDPOBackbone=LLaMA-3-8B2026.02 | 6.88 | 2.25 | |
| rDPOBackbone=Llama-3-8B2026.03 | 6.88 | 2.25 | |
| IPOBackbone=LLaMA-3.2-3B2026.02 | 12.1 | 5.26 | |
| Llama-3.2-3B + IPOBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 12.1 | 5.26 | |
| DPOBackbone=LLaMA-3.2-3B2026.02 | 12.12 | 5.45 | |
| Llama-3.2-3B + DPOBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 12.12 | 5.45 | |
| ShaPO-RBackbone=Pythia-2.8B2026.02 | 12.39 | 2.72 | |
| Pythia-2.8B + wDPOBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 12.95 | 2.25 | |
| ShaPO-TBackbone=Pythia-2.8B2026.02 | 14.14 | 1.67 | |
| Pythia-2.8B + Dr.DPOBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 16.77 | 4.76 | |
| DPOBackbone=Qwen2.5-7B2026.02 | 17.7 | 4.9 | |
| DPOBackbone=Qwen-2.5-7B2026.03 | 17.7 | 4.9 | |
| IPOBackbone=Qwen2.5-7B2026.02 | 18.02 | 5.03 | |
| IPOBackbone=Qwen-2.5-7B2026.03 | 18.02 | 5.03 | |
| IPOBackbone=LLaMA-3-8B2026.02 | 21.7 | 8.3 | |
| IPOBackbone=Llama-3-8B2026.03 | 21.7 | 8.3 | |
| DPOBackbone=LLaMA-3-8B2026.02 | 23.05 | 10.3 | |
| DPOBackbone=Llama-3-8B2026.03 | 23.05 | 10.3 | |
| cDPOBackbone=LLaMA-3.2-3B2026.02 | 24.55 | 10.67 | |
| Llama-3.2-3B + cDPOBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 24.55 | 10.67 | |
| cDPOBackbone=Qwen2.5-7B2026.02 | 26.13 | 8.59 | |
| cDPOBackbone=Qwen-2.5-7B2026.03 | 26.13 | 8.59 | |
| Dr.DPOBackbone=Pythia-2.8B2026.02 | 29.35 | 19.94 | |
| cDPOBackbone=LLaMA-3-8B2026.02 | 30.32 | 13.34 | |
| cDPOBackbone=Llama-3-8B2026.03 | 30.32 | 13.34 | |
| VanillaBackbone=Qwen2.5-7B2026.02 | 32.52 | 10.76 | |
| Qwen-2.5-7BBackbone=Qwen-2.5-7B2026.03 | 32.52 | 10.76 | |
| rDPOBackbone=Pythia-2.8B2026.02 | 55.19 | 21.95 | |
| Pythia-2.8B + rDPOBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 55.19 | 21.95 | |
| DPOBackbone=Pythia-2.8B2026.02 | 58.82 | 21.06 | |
| Pythia-2.8B + DPOBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 58.82 | 21.06 | |
| IPOBackbone=Pythia-2.8B2026.02 | 59.49 | 22.78 | |
| Pythia-2.8B + IPOBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 59.49 | 22.78 | |
| cDPOBackbone=Pythia-2.8B2026.02 | 60.33 | 18.08 | |
| Pythia-2.8B + cDPOBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 60.33 | 18.08 | |
| VanillaBackbone=LLaMA-3-8B2026.02 | 65.1 | 19.92 | |
| Llama-3-8BBackbone=Llama-3-8B2026.03 | 65.1 | 19.92 | |
| SFTBackbone=LLaMA-3.2-3B2026.02 | 66.93 | 38.67 | |
| Llama-3.2-3B + SFTBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 66.93 | 38.67 | |
| VallinaBackbone=LLaMA-3.2-3B2026.02 | 67.61 | 18.89 | |
| Llama-3.2-3BBackbone=Llama-3.2-3B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 67.61 | 18.89 | |
| SFTBackbone=LLaMA-3-8B2026.02 | 69.27 | 47.54 | |
| SFTBackbone=Llama-3-8B2026.03 | 69.27 | 47.54 | |
| SFTBackbone=Qwen2.5-7B2026.02 | 70.54 | 48.01 | |
| SFTBackbone=Qwen-2.5-7B2026.03 | 70.54 | 48.01 | |
| VallinaBackbone=Pythia-2.8B2026.02 | 72.02 | 24.95 | |
| Pythia-2.8BBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 72.02 | 24.95 | |
| SFTBackbone=Pythia-2.8B2026.02 | 72.21 | 25.75 | |
| Pythia-2.8B + SFTBackbone=Pythia-2.8B, Training Dataset=PKU-SafeRLHF-30K2026.03 | 72.21 | 25.75 |