Preference Alignment on HH-RLHF
89.5Win RateCW-IPO
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| CW-IPOAlignment Algorithm=IPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 89.5 | — | — | — | — | — | — | — | — | — | |
| CW-rDPOAlignment Algorithm=rDPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 89.4 | — | — | — | — | — | — | — | — | — | |
| HumanAlignment Algorithm=IPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 86.7 | — | — | — | — | — | — | — | — | — | |
| WS-DPOAlignment Algorithm=IPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 86.6 | — | — | — | — | — | — | — | — | — | |
| WS-DPOAlignment Algorithm=rDPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 85.8 | — | — | — | — | — | — | — | — | — | |
| HumanAlignment Algorithm=rDPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 85 | — | — | — | — | — | — | — | — | — | |
| SEA2025.05 | 83.33 | — | — | — | — | — | — | — | — | — | |
| WS-DPOAlignment Algorithm=DPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 80.7 | — | — | — | — | — | — | — | — | — | |
| Re-Control2025.05 | 80 | — | — | — | — | — | — | — | — | — | |
| CW-DPOAlignment Algorithm=DPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 79.7 | — | — | — | — | — | — | — | — | — | |
| Greedy SFTModel=Llama 1B, Training Protocol=full finetuning2025.09 | 76 | — | — | -1.91 | — | — | — | — | — | — | |
| Greedy SFTModel=Llama 3B, Training Protocol=full finetuning2025.09 | 76 | — | — | -1.89 | — | — | — | — | — | — | |
| HumanAlignment Algorithm=DPO, Model Pair=Qwen2.5-0.5B → Qwen2.5-14B2026.03 | 75.9 | — | — | — | — | — | — | — | — | — | |
| CBS2025.05 | 75 | — | — | — | — | — | — | — | — | — | |
| Bo2Model=Llama 1B, Training Protocol=full finetuning2025.09 | 71 | — | — | -0.75 | — | — | — | — | — | — | |
| GEOALIGNBackbone=Qwen3-1.7B, Vs Baseline=BASE-GRPO2026.06 | 68.36 | — | — | — | — | — | 10.83 | 20.82 | — | — | |
| SFT2025.05 | 66.67 | — | — | — | — | — | — | — | — | — | |
| GEOALIGNBackbone=Qwen3-1.7B, Vs Baseline=PODS2026.06 | 65.07 | — | — | — | — | — | 16.42 | 18.5 | — | — | |
| Greedy DPOModel=Llama 1B, Training Protocol=full finetuning2025.09 | 64 | — | — | -0.63 | — | — | — | — | — | — | |
| DPO-HPSComparison Baseline=SFT, setting=fine-tuning2025.02 | 63.875 | — | — | — | — | — | 13.225 | 22.9 | — | — | |
| GEOALIGNBackbone=Qwen3-1.7B, Vs Baseline=PAR2026.06 | 62.38 | — | — | — | — | — | 14.12 | 23.5 | — | — | |
| GEOALIGNBackbone=Qwen3-1.7B, Vs Baseline=PF-PPO (BW)2026.06 | 61.71 | — | — | — | — | — | 11.45 | 26.84 | — | — | |
| GEOALIGNBackbone=Qwen3-1.7B, Vs Baseline=SEED-GRPO2026.06 | 61.53 | — | — | — | — | — | 12.04 | 26.43 | — | — | |
| GEOALIGNBackbone=Qwen3-4B, Vs Baseline=PODS2026.06 | 59.2 | — | — | — | — | — | 19.72 | 21.08 | — | — | |
| DPO-HPSComparison Baseline=DPO-BT, setting=fine-tuning2025.02 | 58.712 | — | — | — | — | — | 13.6 | 27.687 | — | — | |
| DPO-HPSComparison Baseline=DPO-PL, setting=fine-tuning2025.02 | 57.612 | — | — | — | — | — | 13.875 | 28.513 | — | — | |
| Bo2Model=Llama 3B, Training Protocol=full finetuning2025.09 | 57 | — | — | 0.09 | — | — | — | — | — | — | |
| CW-rDPOAlignment Algorithm=rDPO, Model Pair=OPT-125M → OPT-13B2026.03 | 55.1 | — | — | — | — | — | — | — | — | — | |
| Greedy DPOModel=Llama 3B, Training Protocol=full finetuning2025.09 | 54 | — | — | 0.18 | — | — | — | — | — | — | |
| GEOALIGNBackbone=Qwen3-1.7B, Vs Baseline=PF-PPO (BR)2026.06 | 53.87 | — | — | — | — | — | 14.27 | 31.87 | — | — | |
| CW-DPOAlignment Algorithm=DPO, Model Pair=OPT-125M → OPT-13B2026.03 | 51.9 | — | — | — | — | — | — | — | — | — | |
| HumanAlignment Algorithm=rDPO, Model Pair=OPT-125M → OPT-13B2026.03 | 46.4 | — | — | — | — | — | — | — | — | — | |
| GEOALIGNBackbone=Qwen3-4B, Vs Baseline=PF-PPO (BW)2026.06 | 45.95 | — | — | — | — | — | 27.73 | 26.32 | — | — | |
| GEOALIGNBackbone=Qwen3-4B, Vs Baseline=SEED-GRPO2026.06 | 45.95 | — | — | — | — | — | 27.73 | 26.32 | — | — | |
| CW-IPOAlignment Algorithm=IPO, Model Pair=OPT-125M → OPT-13B2026.03 | 45.8 | — | — | — | — | — | — | — | — | — | |
| GEOALIGNBackbone=Qwen3-4B, Vs Baseline=BASE-GRPO2026.06 | 45.79 | — | — | — | — | — | 27.32 | 26.89 | — | — | |
| WS-DPOAlignment Algorithm=DPO, Model Pair=OPT-125M → OPT-13B2026.03 | 44.6 | — | — | — | — | — | — | — | — | — | |
| HumanAlignment Algorithm=DPO, Model Pair=OPT-125M → OPT-13B2026.03 | 43.9 | — | — | — | — | — | — | — | — | — | |
| HumanAlignment Algorithm=IPO, Model Pair=OPT-125M → OPT-13B2026.03 | 43.7 | — | — | — | — | — | — | — | — | — | |
| WS-DPOAlignment Algorithm=rDPO, Model Pair=OPT-125M → OPT-13B2026.03 | 42.1 | — | — | — | — | — | — | — | — | — | |
| GEOALIGNBackbone=Qwen3-4B, Vs Baseline=PAR2026.06 | 39.88 | — | — | — | — | — | 31.99 | 28.13 | — | — | |
| GEOALIGNBackbone=Qwen3-4B, Vs Baseline=PF-PPO (BR)2026.06 | 39.67 | — | — | — | — | — | 33.01 | 27.32 | — | — | |
| EFTModel=Llama 3B, Training Protocol=full finetuning2025.09 | 35 | — | — | 0.47 | — | — | — | — | — | — | |
| WS-DPOAlignment Algorithm=IPO, Model Pair=OPT-125M → OPT-13B2026.03 | 33.5 | — | — | — | — | — | — | — | — | — | |
| EFTModel=Llama 1B, Training Protocol=full finetuning2025.09 | 30 | — | — | 0.26 | — | — | — | — | — | — | |
| AADModel=Llama 1B, Training Protocol=full finetuning2025.09 | — | — | — | -0.06 | — | — | — | — | — | — | |
| AADModel=Llama 3B, Training Protocol=full finetuning2025.09 | — | — | — | 0.29 | — | — | — | — | — | — | |
| BEEARAttack=BadEdit2026.04 | — | — | — | — | — | — | — | — | 4.2 | 85.9 | |
| BEEARAttack=VPI2026.04 | — | — | — | — | — | — | — | — | 5.5 | 85.7 | |
| BEEARAttack=BadChain2026.04 | — | — | — | — | — | — | — | — | 5.2 | 85.1 | |
| BEEARAttack=EmbedX2026.04 | — | — | — | — | — | — | — | — | 14.1 | 85.2 | |
| CROWAttack=BadEdit2026.04 | — | — | — | — | — | — | — | — | 4.9 | 86.1 | |
| CROWAttack=VPI2026.04 | — | — | — | — | — | — | — | — | 6.2 | 86 | |
| CROWAttack=BadChain2026.04 | — | — | — | — | — | — | — | — | 5.8 | 85.5 | |
| CROWAttack=EmbedX2026.04 | — | — | — | — | — | — | — | — | 14.8 | 85.8 | |
| CW-DPOWeak-to-Strong Model Pair=OPT-125M → OPT-1.3B2026.03 | — | 69.9 | — | — | — | — | — | — | — | — | |
| CW-DPOWeak-to-Strong Model Pair=Qwen2.5-0.5B → Qwen2.5-7B2026.03 | — | 75.2 | — | — | — | — | — | — | — | — | |
| DemoDefenseAttack=BadEdit2026.04 | — | — | — | — | — | — | — | — | 15.1 | 85.8 | |
| DemoDefenseAttack=VPI2026.04 | — | — | — | — | — | — | — | — | 5.5 | 86.9 | |
| DemoDefenseAttack=BadChain2026.04 | — | — | — | — | — | — | — | — | 13.4 | 85.5 | |
| DemoDefenseAttack=EmbedX2026.04 | — | — | — | — | — | — | — | — | 17.4 | 86.8 | |
| DPO-BTBase Algorithm=DPO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | — | — | 0.219 | 0.437 | -5.053 | -5.736 | — | — | — | — | |
| DPO-BTsetting=fine-tuning2025.02 | — | — | 0.23 | 0.431 | 0.349 | -0.455 | — | — | — | — | |
| DPO-HPSBase Algorithm=DPO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | — | — | 0.219 | 0.438 | -4.816 | -5.499 | — | — | — | — | |
| DPO-HPSsetting=fine-tuning2025.02 | — | — | 0.232 | 0.43 | 2.723 | 2.04 | — | — | — | — | |
| DPO-PLBase Algorithm=DPO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | — | — | 0.219 | 0.435 | -5.122 | -5.638 | — | — | — | — | |
| DPO-PLsetting=fine-tuning2025.02 | — | — | 0.23 | 0.43 | -0.795 | -1.448 | — | — | — | — | |
| EXO-BTBase Algorithm=EXO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | — | — | 0.203 | 0.442 | -4.825 | -5.508 | — | — | — | — | |
| EXO-BTsetting=fine-tuning2025.02 | — | — | 0.231 | 0.43 | 0.816 | 0.215 | — | — | — | — | |
| EXO-HPSBase Algorithm=EXO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | — | — | 0.197 | 0.444 | -4.583 | -5.266 | — | — | — | — | |
| EXO-HPSsetting=fine-tuning2025.02 | — | — | 0.232 | 0.432 | 1.079 | 0.41 | — | — | — | — | |
| EXO-PLBase Algorithm=EXO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | — | — | 0.222 | 0.436 | -5.183 | -6.166 | — | — | — | — | |
| EXO-PLsetting=fine-tuning2025.02 | — | — | 0.232 | 0.432 | -0.724 | -1.406 | — | — | — | — | |
| FABEAttack=BadEdit2026.04 | — | — | — | — | — | — | — | — | 18.2 | 84.1 | |
| FABEAttack=VPI2026.04 | — | — | — | — | — | — | — | — | 14.5 | 83.8 | |
| FABEAttack=BadChain2026.04 | — | — | — | — | — | — | — | — | 14.2 | 83.2 | |
| FABEAttack=EmbedX2026.04 | — | — | — | — | — | — | — | — | 19.5 | 83.5 | |
| Human (100%)Weak-to-Strong Model Pair=OPT-125M → OPT-1.3B2026.03 | — | 71.9 | — | — | — | — | — | — | — | — | |
| Human (100%)Weak-to-Strong Model Pair=Qwen2.5-0.5B → Qwen2.5-7B2026.03 | — | 72.7 | — | — | — | — | — | — | — | — | |
| IPO-BTBase Algorithm=IPO, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | — | — | 0.189 | 0.446 | -21.821 | -22.504 | — | — | — | — | |
| IPO-BTsetting=fine-tuning2025.02 | — | — | 0.232 | 0.428 | -0.382 | -1.254 | — | — | — | — | |
| IPO-HPSBase Algorithm=IPO, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | — | — | 0.187 | 0.449 | -20.938 | -21.255 | — | — | — | — | |
| IPO-HPSsetting=fine-tuning2025.02 | — | — | 0.231 | 0.424 | -0.321 | -0.926 | — | — | — | — | |
| IPO-PLBase Algorithm=IPO, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | — | — | 0.189 | 0.439 | -60.129 | -60.809 | — | — | — | — | |
| IPO-PLsetting=fine-tuning2025.02 | — | — | 0.223 | 0.429 | -5.199 | -5.264 | — | — | — | — | |
| LLMBDAttack=BadEdit2026.04 | — | — | — | — | — | — | — | — | 16.5 | 82.4 | |
| LLMBDAttack=VPI2026.04 | — | — | — | — | — | — | — | — | 12.2 | 82.1 | |
| LLMBDAttack=BadChain2026.04 | — | — | — | — | — | — | — | — | 11.9 | 86 | |
| LLMBDAttack=EmbedX2026.04 | — | — | — | — | — | — | — | — | 18.8 | 82 | |
| NCA-BTBase Algorithm=NCA, Sampling Strategy=BT, Setting=Transfer Learning2025.02 | — | — | 0.226 | 0.435 | -4.673 | -5.557 | — | — | — | — | |
| NCA-BTsetting=fine-tuning2025.02 | — | — | 0.229 | 0.432 | -1.702 | -3.121 | — | — | — | — | |
| NCA-HPSBase Algorithm=NCA, Sampling Strategy=HPS, Setting=Transfer Learning2025.02 | — | — | 0.224 | 0.436 | -5.378 | -5.562 | — | — | — | — | |
| NCA-HPSsetting=fine-tuning2025.02 | — | — | 0.231 | 0.432 | -0.822 | -1.268 | — | — | — | — | |
| NCA-PLBase Algorithm=NCA, Sampling Strategy=PL, Setting=Transfer Learning2025.02 | — | — | 0.214 | 0.433 | -61.084 | -61.762 | — | — | — | — | |
| NCA-PLsetting=fine-tuning2025.02 | — | — | 0.221 | 0.431 | -5.76 | -5.819 | — | — | — | — | |
| No Def.Attack=BadEdit2026.04 | — | — | — | — | — | — | — | — | 99.4 | 88.2 | |
| No Def.Attack=VPI2026.04 | — | — | — | — | — | — | — | — | 96.5 | 88.8 | |
| No Def.Attack=BadChain2026.04 | — | — | — | — | — | — | — | — | 95.1 | 87.9 | |
| No Def.Attack=EmbedX2026.04 | — | — | — | — | — | — | — | — | 98.1 | 88.4 | |
| SFT Modelsetting=fine-tuning2025.02 | — | — | 0.22 | 0.425 | — | — | — | — | — | — |