Human Preference Alignment on PKU-SafeRLHF
0.324BLEUEXO-BT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| EXO-BTsetting=fine-tuning2025.02 | 0.324 | 0.421 | -5.553 | -6.164 | |
| EXO-HPSsetting=fine-tuning2025.02 | 0.314 | 0.425 | -5.495 | -6.031 | |
| DPO-HPSBase Algorithm=DPO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | 0.31 | 0.407 | 5.725 | 5.116 | |
| EXO-HPSBase Algorithm=EXO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | 0.31 | 0.407 | 2.903 | 3.495 | |
| IPO-BTsetting=fine-tuning2025.02 | 0.31 | 0.405 | -23.07 | -23.678 | |
| SPPO-PLBase Algorithm=SPPO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | 0.309 | 0.406 | -8.16 | -8.197 | |
| SPPO-BTBase Algorithm=SPPO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | 0.309 | 0.407 | -0.134 | -0.843 | |
| SPPO-HPSBase Algorithm=SPPO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | 0.309 | 0.407 | -0.101 | -0.81 | |
| NCA-BTBase Algorithm=NCA, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | 0.309 | 0.407 | -0.373 | -0.982 | |
| IPO-PLsetting=fine-tuning2025.02 | 0.309 | 0.401 | -66.337 | -66.946 | |
| DPO-BTBase Algorithm=DPO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | 0.308 | 0.407 | 1.346 | 1.738 | |
| NCA-HPSBase Algorithm=NCA, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | 0.308 | 0.407 | -0.102 | -0.711 | |
| IPO-HPSsetting=fine-tuning2025.02 | 0.308 | 0.406 | -21.607 | -22.215 | |
| DPO-PLBase Algorithm=DPO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | 0.307 | 0.407 | 0.144 | -0.464 | |
| EXO-BTBase Algorithm=EXO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | 0.306 | 0.407 | 2.71 | 2.102 | |
| IPO-PLBase Algorithm=IPO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | 0.306 | 0.406 | 2.294 | 1.686 | |
| IPO-BTBase Algorithm=IPO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | 0.306 | 0.405 | 4.393 | 3.786 | |
| NCA-PLBase Algorithm=NCA, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | 0.306 | 0.409 | -8.23 | -8.267 | |
| DPO-BTsetting=fine-tuning2025.02 | 0.306 | 0.417 | -5.441 | -6.167 | |
| DPO-HPSsetting=fine-tuning2025.02 | 0.306 | 0.407 | -5.359 | -5.851 | |
| IPO-HPSBase Algorithm=IPO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | 0.305 | 0.409 | 5.386 | 4.779 | |
| DPO-PLsetting=fine-tuning2025.02 | 0.305 | 0.412 | -6.852 | -5.961 | |
| NCA-BTsetting=fine-tuning2025.02 | 0.305 | 0.41 | -5.135 | -5.644 | |
| NCA-HPSsetting=fine-tuning2025.02 | 0.304 | 0.411 | -5.109 | -5.318 | |
| EXO-PLBase Algorithm=EXO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | 0.303 | 0.409 | 0.845 | 0.237 | |
| EXO-PLsetting=fine-tuning2025.02 | 0.303 | 0.409 | -5.455 | -6.128 | |
| NCA-PLsetting=fine-tuning2025.02 | 0.3 | 0.411 | -70.91 | -71.518 | |
| SPPO-HPSsetting=fine-tuning2025.02 | 0.298 | 0.435 | -5.273 | -5.881 | |
| SPPO-PLsetting=fine-tuning2025.02 | 0.297 | 0.413 | -67.474 | -68.082 | |
| SPPO-BTsetting=fine-tuning2025.02 | 0.297 | 0.433 | -13.442 | -14.05 | |
| SFT Modelsetting=fine-tuning2025.02 | 0.294 | 0.406 | — | — |