Preference Evaluation on AlpacaEval 2
559WR (%)SelectiveDPO
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| SelectiveDPOBase Model=Qwen-2.5-7B, Training Dataset=UltraFeedback2026.02 | 559 | — | 312 | — | |
| MixDPOBase Model=Qwen-2.5-7B, Training Dataset=UltraFeedback2026.02 | 559 | — | 345 | — | |
| VANILLA DPOBase Model=Qwen-2.5-7B, Training Dataset=UltraFeedback2026.02 | 360 | — | 238 | — | |
| SimPOBase Model=Qwen-2.5-7B, Training Dataset=UltraFeedback2026.02 | 311 | — | 228 | — | |
| BASE SFTBase Model=Qwen-2.5-7B, Training Dataset=UltraFeedback2026.02 | 99 | — | 20 | — | |
| GRPO + SAVEPolicy Model=Qwen3-4B-Instruct-25072026.05 | 62.69 | 53.28 | — | — | |
| GRPO + Improved RMPolicy Model=Qwen3-4B-Instruct-25072026.05 | 62.4 | 54.24 | — | — | |
| GRPO + R2MPolicy Model=Qwen3-4B-Instruct-25072026.05 | 62.26 | 52 | — | — | |
| RePOBackbone=Qwen3-4B-Base2026.06 | 60.12 | 55.08 | — | — | |
| GRPOPolicy Model=Qwen3-4B-Instruct-25072026.05 | 59.81 | 51.68 | — | — | |
| SFTPolicy Model=Qwen3-4B-Instruct-25072026.05 | 58.59 | 50.01 | — | — | |
| REINFORCE++Policy Model=Qwen3-4B-Instruct-25072026.05 | 55.81 | 49.04 | — | — | |
| KTOBackbone=Qwen3-4B-Base2026.06 | 55.78 | 52.31 | — | — | |
| RePO_detBackbone=Qwen3-4B-Base2026.06 | 55.53 | 51.66 | — | — | |
| PRIMEPolicy Model=Qwen3-4B-Instruct-25072026.05 | 52.83 | 43.37 | — | — | |
| RePOBackbone=Qwen3-1.7B-Base2026.06 | 43.66 | 36.61 | — | — | |
| RePO_detBackbone=Qwen3-1.7B-Base2026.06 | 41.42 | 34.95 | — | — | |
| Vanilla RMPolicy=Best-of-N2026.04 | 40.99 | 48.31 | — | 1,667 | |
| CARPPolicy=Best-of-N2026.04 | 40.99 | 49.94 | — | 1,611 | |
| KTOBackbone=Qwen3-1.7B-Base2026.06 | 38.93 | 34.73 | — | — | |
| IPOBackbone=Qwen3-4B-Base2026.06 | 38.63 | 36.43 | — | — | |
| RAG-PrefOffline Alignment=RTO2026.05 | 34.29 | 37.45 | — | — | |
| DPOBackbone=Qwen3-4B-Base2026.06 | 33.92 | 32.89 | — | — | |
| RAGOffline Alignment=RTO2026.05 | 32.67 | 36.56 | — | — | |
| MATRIX-Gen-DPOBase LLM=MATRIX-SFT-Model2024.10 | 31.3 | 24.2 | — | — | |
| RTO (Offline Baseline)Online Alignment=None2026.05 | 31.3 | 36.17 | — | — | |
| RPOBackbone=Qwen3-4B-Base2026.06 | 28.23 | 29.51 | — | — | |
| GRPO + Improved RMPolicy Model=Qwen2.5-3B-Instruct2026.05 | 27.72 | 20.2 | — | — | |
| IPOBackbone=Qwen3-1.7B-Base2026.06 | 26.37 | 24.46 | — | — | |
| DPOBackbone=Qwen3-1.7B-Base2026.06 | 25.84 | 23.9 | — | — | |
| GRPOPolicy Model=Qwen2.5-3B-Instruct2026.05 | 24.41 | 16.31 | — | — | |
| Llama-3-8B-InstructNote=Reference Model2024.10 | 22.57 | 22.92 | — | — | |
| GRPO + SAVEPolicy Model=Qwen2.5-3B-Instruct2026.05 | 21.59 | 17.85 | — | — | |
| GRPO + R2MPolicy Model=Qwen2.5-3B-Instruct2026.05 | 21.05 | 17.45 | — | — | |
| Magpie-PRO-DPOBase LLM=MATRIX-SFT-Model2024.10 | 20.3 | 18.99 | — | — | |
| OrcaBase LLM=MATRIX-SFT-Model2024.10 | 20.1 | 17.26 | — | — | |
| RAG-PrefOffline Alignment=DPO2026.05 | 19.5 | 24.82 | — | — | |
| REINFORCE++Policy Model=Qwen2.5-3B-Instruct2026.05 | 19.1 | 15.98 | — | — | |
| RAG-PrefOffline Alignment=PPO2026.05 | 19.07 | 28.59 | — | — | |
| UltraFeedbackBase LLM=MATRIX-SFT-Model2024.10 | 18.48 | 17.17 | — | — | |
| SFTPolicy Model=Qwen2.5-3B-Instruct2026.05 | 18.41 | 15.36 | — | — | |
| RAG-PrefOffline Alignment=SimPO2026.05 | 18.14 | 23.18 | — | — | |
| RAGOffline Alignment=PPO2026.05 | 17.76 | 27.78 | — | — | |
| TDPOBackbone=Qwen3-4B-Base2026.06 | 17.08 | 17.97 | — | — | |
| RAGOffline Alignment=DPO2026.05 | 16.83 | 21.81 | — | — | |
| RPOBackbone=Qwen3-1.7B-Base2026.06 | 15.47 | 18.63 | — | — | |
| RAGOffline Alignment=SimPO2026.05 | 14.66 | 19.66 | — | — | |
| PPO (Offline Baseline)Online Alignment=None2026.05 | 14.66 | 18.3 | — | — | |
| PRIMEPolicy Model=Qwen2.5-3B-Instruct2026.05 | 14.33 | 12.18 | — | — | |
| Tulu-3-DPOBase LLM=MATRIX-SFT-Model2024.10 | 13.79 | 12.91 | — | — | |
| DPO (Offline Baseline)Online Alignment=None2026.05 | 12.86 | 14.46 | — | — | |
| BaseBackbone=Qwen3-4B-Base2026.06 | 11.62 | 12.8 | — | — | |
| ArgillaMixBase LLM=MATRIX-SFT-Model2024.10 | 11.15 | 9.75 | — | — | |
| RAG-PrefOffline Alignment=SFT2026.05 | 10.87 | 14.48 | — | — | |
| TDPOBackbone=Qwen3-1.7B-Base2026.06 | 10.72 | 12.24 | — | — | |
| SimPO (Offline Baseline)Online Alignment=None2026.05 | 10.56 | 16.85 | — | — | |
| RAGOffline Alignment=SFT2026.05 | 10.37 | 13.26 | — | — | |
| OPADOffline Alignment=RTO2026.05 | 9.62 | 10.69 | — | — | |
| SFT (Offline Baseline)Online Alignment=None2026.05 | 9.44 | 14.17 | — | — | |
| OPADOffline Alignment=PPO2026.05 | 7.95 | 10.16 | — | — | |
| OPADOffline Alignment=SimPO2026.05 | 7.95 | 8.41 | — | — | |
| BaseBackbone=Qwen3-1.7B-Base2026.06 | 6.05 | 8.27 | — | — | |
| OPADOffline Alignment=DPO2026.05 | 5.59 | 7.61 | — | — | |
| OPADOffline Alignment=SFT2026.05 | 0.99 | 2.26 | — | — |