Assistant Response Alignment (Helpfulness and Harmlessness) on HH-RLHF (test)
89.42Helpfulness Win RateMetaAligner-7B
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MetaAligner-7BPolicy Model=LLaMA2-Chat-70B, GPU Hours=242.682024.03 | 89.42 | — | 85.16 | 88.08 | 87.55 | — | — | — | — | — | — | |
| MetaAligner-13BPolicy Model=LLaMA2-Chat-7B, GPU Hours=403.442024.03 | 85.5 | — | 76.5 | 86 | 82.67 | — | — | — | — | — | — | |
| Self-RefinementPolicy Model=LLaMA2-Chat-70B2024.03 | 82.8 | — | 70.48 | 68.91 | 74.06 | — | — | — | — | — | — | |
| MetaAligner-7BPolicy Model=LLaMA2-Chat-7B, GPU Hours=242.682024.03 | 82 | — | 77.5 | 83 | 80.83 | — | — | — | — | — | — | |
| MetaAligner-7B (w/o Equal Pref.)Policy Model=LLaMA2-Chat-7B, GPU Hours=242.682024.03 | 80.7 | — | 73.82 | 77.39 | 77.3 | — | — | — | — | — | — | |
| MetaAligner-7B (w/o Warm Up)Policy Model=LLaMA2-Chat-7B, GPU Hours=242.682024.03 | 80.32 | — | 77.1 | 82.63 | 80.02 | — | — | — | — | — | — | |
| RLHF-r1Alignment Algorithm=RLHF, Objective=r1, Base Model=phi-1.52025.12 | 76 | — | — | — | — | — | — | — | — | -0.42 | -0.23 | |
| SFTPolicy Model=LLaMA2-Chat-7B, GPU Hours=247.742024.03 | 75 | — | 72 | 76.5 | 74.5 | — | — | — | — | — | — | |
| Aligner-7BPolicy Model=LLaMA2-Chat-7B, GPU Hours=236.82024.03 | 75 | — | 66.5 | 70.12 | 70.54 | — | — | — | — | — | — | |
| MetaAligner-1.1BPolicy Model=LLaMA2-Chat-7B, GPU Hours=120.482024.03 | 75 | — | 62.5 | 77 | 71.5 | — | — | — | — | — | — | |
| ChatGPTprotocol=zero-shot2023.10 | 68.28 | — | 72.19 | — | 70.43 | 69.85 | 71.02 | — | — | — | — | |
| CycleAlign_PROBackbone=Alpaca2023.10 | 67.89 | — | 71.32 | — | 68.97 | 66.53 | 68.92 | — | — | — | — | |
| CycleAlign_RRHFBackbone=LLaMA2023.10 | 67.05 | — | 71.66 | — | 68.43 | 65.89 | 67.95 | — | — | — | — | |
| CycleAlign_PROBackbone=LLaMA2023.10 | 66.49 | — | 70.62 | — | 68.41 | 67.67 | 68.5 | — | — | — | — | |
| PROBackbone=Alpaca2023.10 | 64.56 | — | 73.13 | — | 67.64 | 65.6 | 66.51 | — | — | — | — | |
| PROBackbone=LLaMA2023.10 | 64.05 | — | 72.86 | — | 67.4 | 65.56 | 66.44 | — | — | — | — | |
| MODPOPolicy Model=LLaMA2-Chat-7B, GPU Hours=405.92024.03 | 64 | — | 65 | 78 | 69 | — | — | — | — | — | — | |
| ChatGLMprotocol=zero-shot, Backbone=ChatGLM-6B2023.10 | 62.14 | — | 67.26 | — | 63.85 | 60.44 | 63.86 | — | — | — | — | |
| RRHFBackbone=LLaMA2023.10 | 61.38 | — | 64.63 | — | 63.12 | 63.26 | 63.28 | — | — | — | — | |
| PPOBackbone=LLaMA2023.10 | 55.29 | — | 61.97 | — | 58.65 | 59.78 | 58.26 | — | — | — | — | |
| Alpacaprotocol=zero-shot, Backbone=Alpaca-7B2023.10 | 53.85 | — | 52.77 | — | 54.26 | 55.3 | 55.43 | — | — | — | — | |
| MORLHFPolicy Model=LLaMA2-Chat-7B, GPU Hours=1892.32024.03 | 51.2 | — | 62.83 | 77.5 | 63.84 | — | — | — | — | — | — | |
| MOPO-LagAlignment Algorithm=MOPO, Variant=Lagrangian (Lag), Base Model=phi-1.52025.12 | 39 | — | — | — | — | — | — | — | — | 0.22 | 0.17 | |
| LLaMAprotocol=zero-shot, Backbone=LLaMA-7B2023.10 | 33.25 | — | 53.59 | — | 40.67 | 40.48 | 36.23 | — | — | — | — | |
| PARMAlignment Algorithm=PARM, Base Model=phi-1.52025.12 | 31 | — | — | — | — | — | — | — | — | 0.17 | 0.23 | |
| MOPO-LBAlignment Algorithm=MOPO, Variant=Lower Bound (LB), Base Model=phi-1.52025.12 | 30 | — | — | — | — | — | — | — | — | 0.19 | 0.18 | |
| RiCAlignment Algorithm=RiC, Base Model=phi-1.52025.12 | 25 | — | — | — | — | — | — | — | — | 0.15 | 0.11 | |
| DPOAlignment Algorithm=DPO, Alignment Dataset=D_J, Base Model=phi-1.52025.12 | 18 | — | — | — | — | — | — | — | — | 0.09 | 0.11 | |
| MODPOAlignment Algorithm=MODPO, Base Model=phi-1.52025.12 | 4 | — | — | — | — | — | — | — | — | -0.09 | 0.08 | |
| RLHF-r3Alignment Algorithm=RLHF, Objective=r3, Base Model=phi-1.52025.12 | -79 | — | — | — | — | — | — | — | — | -0.92 | 0.42 | |
| RLHF-r2Alignment Algorithm=RLHF, Objective=r2, Base Model=phi-1.52025.12 | -81 | — | — | — | — | — | — | — | — | 0.53 | -0.4 | |
| DLMA-7BBase Model=Mistral-7B2024.02 | — | — | — | — | 43 | — | — | 17 | 40 | — | — | |
| DLMA-7BBase Model=Falcon-7B2024.02 | — | — | — | — | 44 | — | — | 15 | 41 | — | — | |
| MAT-STEERcomparison_baseline=ICL2025.02 | — | 67.07 | — | — | — | — | — | — | — | — | — | |
| MAT-STEERcomparison_baseline=SFT2025.02 | — | 62.35 | — | — | — | — | — | — | — | — | — | |
| MAT-STEERcomparison_baseline=LITO2025.02 | — | 59.12 | — | — | — | — | — | — | — | — | — |