Instruction Following and Helpfulness Evaluation on AlpacaEval 2.0
49.4Win RateQWEN2-INSTRUCT w/ SSO_DPO
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| QWEN2-INSTRUCT w/ SSO_DPOBackbone=QWEN2-INSTRUCT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 49.4 | 36.5 | |
| QWEN2-SFT w/ SSO_DPOBackbone=QWEN2-SFT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 45.4 | 43 | |
| QWEN2-INSTRUCT w/ PBAA_DPOBackbone=QWEN2-INSTRUCT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 43.7 | 34.7 | |
| QWEN2-SFT w/ PBAA_DPOBackbone=QWEN2-SFT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 42.5 | 37.7 | |
| LLAMA3-INSTRUCT w/ SSO_DPOBackbone=LLAMA3-INSTRUCT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 28.9 | 25.6 | |
| Off-Policy AlignDistilBackbone=Llama3-8B2025.03 | 28.51 | 24.21 | |
| Qwen2.5-7BModel size=7B2025.06 | 28.34 | — | |
| LLAMA3-SFT w/ SSO_DPOBackbone=LLAMA3-SFT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 28.3 | 35 | |
| On-Policy AlignDistilBackbone=Llama3-8B2025.03 | 27.64 | 26.29 | |
| LLAMA3-INSTRUCT w/ PBAA_DPOBackbone=LLAMA3-INSTRUCT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 25.7 | 23.8 | |
| On-Policy AlignDistilBackbone=Qwen2.5-7B2025.03 | 25.53 | 31.32 | |
| Off-Policy AlignDistilBase Model=Qwen2.5-1.5B-Instruct2025.03 | 24.29 | 21.16 | |
| LLAMA3-SFT w/ PBAA_DPOBackbone=LLAMA3-SFT, Alignment Protocol=Principle-Based Automated Alignment, Synthetic Data=true2024.10 | 24 | 29.5 | |
| Off-Policy AlignDistilBackbone=Qwen2.5-7B2025.03 | 23.82 | 29.16 | |
| LLAMA3-INSTRUCT w/ ULTRAFEEDBACKBackbone=LLAMA3-INSTRUCT, Synthetic Data=false2024.10 | 22.6 | 23.8 | |
| On-Policy AlignDistilBase Model=Qwen2.5-1.5B-Instruct2025.03 | 22.11 | 19.45 | |
| TDPO1Backbone=Qwen2.5-7B2025.03 | 19.44 | 26.42 | |
| QWEN2-INSTRUCT w/ ULTRAFEEDBACKBackbone=QWEN2-INSTRUCT, Synthetic Data=false2024.10 | 17.6 | 20.4 | |
| LLAMA3-SFT w/ ULTRAFEEDBACKBackbone=LLAMA3-SFT, Synthetic Data=false2024.10 | 17.5 | 22 | |
| TDPO2Backbone=Qwen2.5-7B2025.03 | 17.33 | 22.63 | |
| QWEN2-INSTRUCTBackbone=QWEN2-INSTRUCT2024.10 | 17.2 | 19.5 | |
| DPOBackbone=Qwen2.5-7B2025.03 | 17.1 | 22.25 | |
| RTOBackbone=Llama3-8B2025.03 | 16.91 | 17 | |
| Qwen2.5-1.5B + daDPOBase model=Qwen2.5-1.5B, RLHF method=daDPO2025.06 | 16.41 | — | |
| On-Policy AlignDistilBase Model=Qwen2-1.5B-Instruct2025.03 | 15.65 | 12.93 | |
| DPOBackbone=Llama3-8B2025.03 | 15.58 | 15.67 | |
| RTOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 15.53 | 16.54 | |
| LLAMA3-SFTBackbone=LLAMA3-SFT2024.10 | 15 | 20.6 | |
| QWEN2-SFT w/ ULTRAFEEDBACKBackbone=QWEN2-SFT, Synthetic Data=false2024.10 | 15 | 20.2 | |
| Qwen2.5-1.5B + dDPOBase model=Qwen2.5-1.5B, RLHF method=dDPO2025.06 | 14.66 | — | |
| Off-Policy AlignDistilBase Model=Qwen2-1.5B-Instruct2025.03 | 14.29 | 11.79 | |
| DPO beta=0.01Base Model=Qwen2.5-1.5B-Instruct, beta=0.012025.03 | 14.29 | 14.09 | |
| TDPO2Backbone=Llama3-8B2025.03 | 13.79 | 14.48 | |
| TDPO1Backbone=Llama3-8B2025.03 | 13.73 | 14.53 | |
| QWEN2-SFTBackbone=QWEN2-SFT2024.10 | 13.2 | 20.1 | |
| LLAMA3-INSTRUCTBackbone=LLAMA3-INSTRUCT2024.10 | 13.2 | 20.1 | |
| SimPOBackbone=Llama3-8B2025.03 | 12.92 | 11.24 | |
| PPOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 12.67 | 18.06 | |
| RTOBackbone=Qwen2.5-7B2025.03 | 12 | 17.75 | |
| DPO beta=0.01Base Model=Qwen2-1.5B-Instruct, beta=0.012025.03 | 11.61 | 10.72 | |
| DPOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 10.74 | 14.35 | |
| SimPOBackbone=Qwen2.5-7B2025.03 | 10.37 | 15.65 | |
| KTOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 10 | 14.07 | |
| TDPO1Base Model=Qwen2.5-1.5B-Instruct2025.03 | 9.94 | 13.19 | |
| SimPOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 9.81 | 11.61 | |
| SimPOBase Model=Qwen2-1.5B-Instruct2025.03 | 9.63 | 8.19 | |
| RTOBase Model=Qwen2-1.5B-Instruct2025.03 | 9.32 | 8.92 | |
| TDPO2Base Model=Qwen2.5-1.5B-Instruct2025.03 | 9.07 | 13.64 | |
| Initial ModelBase Model=Qwen2.5-1.5B-Instruct2025.03 | 8.94 | 12.57 | |
| Qwen2.5-1.5BModel size=1.5B2025.06 | 6.44 | — | |
| KTOBase Model=Qwen2-1.5B-Instruct2025.03 | 6.34 | 7.16 | |
| DPOBase Model=Qwen2-1.5B-Instruct2025.03 | 5.03 | 6.42 | |
| TDPO1Base Model=Qwen2-1.5B-Instruct2025.03 | 4.6 | 6.58 | |
| PPOBase Model=Qwen2-1.5B-Instruct2025.03 | 4.41 | 4.86 | |
| SFTBackbone=Qwen2.5-7B2025.03 | 3.66 | 7.51 | |
| TDPO2Base Model=Qwen2-1.5B-Instruct2025.03 | 2.42 | 3.59 | |
| SFTBackbone=Llama3-8B2025.03 | 2.3 | 3 | |
| Initial ModelBase Model=Qwen2-1.5B-Instruct2025.03 | 1.99 | 3.1 |