Safety Alignment on PKU-SafeRLHF
3.92Gold RewardPPO
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| PPO2025.12 | 3.92 | 1.11 | 0.042 | |
| RSOOracle=Model2025.12 | 2.84 | 1.14 | 0.037 | |
| Randomdelta (δ)=1.5, Oracle=Model2025.12 | 2.29 | 0.37 | 0.18 | |
| Randomdelta (δ)=1.4, Oracle=Model2025.12 | 2.27 | 0.063 | 0.0007 | |
| Randomdelta (δ)=1.4, Oracle=GPT-4o2025.12 | 2.04 | 0.14 | -0.031 | |
| Randomdelta (δ)=1.5, Oracle=GPT-4o2025.12 | 1.89 | 0.59 | 0.031 | |
| SHF-CASdelta (δ)=1.5, Oracle=Model2025.12 | 1.6 | 0.036 | 0.32 | |
| SHF-CASdelta (δ)=1.4, Oracle=Model2025.12 | 1.59 | 0.0003 | 0.0019 | |
| SHF-CASdelta (δ)=1.4, Oracle=GPT-4o2025.12 | 1.31 | 0.0041 | 0.0007 | |
| SHF-CASdelta (δ)=1.5, Oracle=GPT-4o2025.12 | 1.21 | 0.067 | 0.1 | |
| Randomdelta (δ)=1.6, Oracle=Model2025.12 | -0.64 | 1.35 | 0.026 | |
| Randomdelta (δ)=1.6, Oracle=GPT-4o2025.12 | -1.46 | 1.41 | 0.017 | |
| SHF-CASdelta (δ)=1.6, Oracle=Model2025.12 | -2.9 | 0.16 | 0.34 | |
| SHF-CASdelta (δ)=1.6, Oracle=GPT-4o2025.12 | -3.53 | 0.35 | 0.11 |