LLM Alignment Evaluation on Arena-Hard
42.7Win RateAAO
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AAOBackbone=Llama3.1-8B-Instruct2025.11 | 42.7 | — | |
| AAOBackbone=Llama3.1-8B-Base2025.11 | 41 | — | |
| PSFT → DPOBackbone=Qwen3-4B-Base, Training Dataset=UltraFeedback, Post-training=DPO2025.08 | 36.4 | — | |
| SimPOBackbone=Llama3.1-8B-Instruct2025.11 | 33.5 | — | |
| SimPOBackbone=Llama3.1-8B-Base2025.11 | 30.6 | — | |
| RTOBackbone=Llama3.1-8B-Instruct2025.11 | 30.4 | — | |
| PSFTprolong → DPOBackbone=Qwen3-4B-Base, Training Dataset=UltraFeedback, Post-training=DPO2025.08 | 30.2 | — | |
| TIS-DPOBackbone=Llama3.1-8B-Instruct2025.11 | 30.1 | — | |
| DPOBackbone=Llama3.1-8B-Instruct2025.11 | 29.6 | — | |
| IPOBackbone=Llama3.1-8B-Instruct2025.11 | 28.3 | — | |
| DPO-RandomBackbone=Llama3.1-8B-Instruct2025.11 | 27.9 | — | |
| TIS-DPOBackbone=Llama3.1-8B-Base2025.11 | 27.8 | — | |
| TDPOBackbone=Llama3.1-8B-Base2025.11 | 27.2 | — | |
| KTOBackbone=Llama3.1-8B-Instruct2025.11 | 26.8 | — | |
| RTOBackbone=Llama3.1-8B-Base2025.11 | 26.7 | — | |
| SFT → DPOBackbone=Qwen3-4B-Base, Training Dataset=UltraFeedback, Post-training=DPO2025.08 | 26.5 | — | |
| TDPOBackbone=Llama3.1-8B-Instruct2025.11 | 26.4 | — | |
| DPOBackbone=Llama3.1-8B-Base2025.11 | 26 | — | |
| KTOBackbone=Llama3.1-8B-Base2025.11 | 25.8 | — | |
| DPO-RandomBackbone=Llama3.1-8B-Base2025.11 | 25.4 | — | |
| IPOBackbone=Llama3.1-8B-Base2025.11 | 25.1 | — | |
| AAOBackbone=Mistral-7B-Instruct2025.11 | 24.2 | — | |
| Off-Policy AlignDistilBase Model=Qwen2.5-1.5B-Instruct2025.03 | 24.1 | 21.8 | |
| On-Policy AlignDistilBase Model=Qwen2.5-1.5B-Instruct2025.03 | 24 | 23 | |
| MATRIX-Gen-DPOBase LLM=MATRIX-SFT-Model2024.10 | 22.7 | — | |
| SimPOBackbone=Mistral-7B-Instruct2025.11 | 22.6 | — | |
| Llama-3-8B-InstructNote=Reference Model2024.10 | 20.6 | — | |
| KTOBackbone=Mistral-7B-Instruct2025.11 | 18.6 | — | |
| TDPOBackbone=Mistral-7B-Instruct2025.11 | 18.3 | — | |
| RTOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 18.2 | 16.6 | |
| TIS-DPOBackbone=Mistral-7B-Instruct2025.11 | 18 | — | |
| RTOBackbone=Mistral-7B-Instruct2025.11 | 17.9 | — | |
| SFTBackbone=Qwen3-4B-Base, Training Dataset=UltraFeedback2025.08 | 17.9 | — | |
| PSFTprolongBackbone=Qwen3-4B-Base, Training Dataset=UltraFeedback2025.08 | 17.4 | — | |
| DPOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 17.1 | 14.8 | |
| DPOBackbone=Mistral-7B-Instruct2025.11 | 17 | — | |
| Initial ModelBase Model=Qwen2.5-1.5B-Instruct2025.03 | 16.8 | 12.7 | |
| IPOBackbone=Mistral-7B-Instruct2025.11 | 16.8 | — | |
| DPO-RandomBackbone=Mistral-7B-Instruct2025.11 | 16.8 | — | |
| TDPO1Base Model=Qwen2.5-1.5B-Instruct2025.03 | 16.5 | 14.1 | |
| DPO beta=0.01Base Model=Qwen2.5-1.5B-Instruct, beta=0.012025.03 | 16.2 | 15.5 | |
| PPOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 15.9 | 13.7 | |
| Magpie-PRO-DPOBase LLM=MATRIX-SFT-Model2024.10 | 15.9 | — | |
| TDPO2Base Model=Qwen2.5-1.5B-Instruct2025.03 | 15.8 | 13 | |
| OrcaBase LLM=MATRIX-SFT-Model2024.10 | 15.2 | — | |
| KTOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 15 | 12.3 | |
| UltraFeedbackBase LLM=MATRIX-SFT-Model2024.10 | 14 | — | |
| PUGCModel=PUGC+DPO, Alignment=DPO2025.06 | 13.7 | — | |
| AAOBackbone=Mistral-7B-Base2025.11 | 13.6 | — | |
| Tulu-3-DPOBase LLM=MATRIX-SFT-Model2024.10 | 13.1 | — | |
| Mistral-7B-instructModel=Mistral-7B-instruct2025.06 | 12.6 | — | |
| PSFTBackbone=Qwen3-4B-Base, Training Dataset=UltraFeedback2025.08 | 11.4 | — | |
| ArgillaMixBase LLM=MATRIX-SFT-Model2024.10 | 11.3 | — | |
| On-Policy AlignDistilBase Model=Qwen2-1.5B-Instruct2025.03 | 11 | 6.7 | |
| Off-Policy AlignDistilBase Model=Qwen2-1.5B-Instruct2025.03 | 10.5 | 6 | |
| SimPOBackbone=Mistral-7B-Base2025.11 | 9.7 | — | |
| DPO-RandomBackbone=Mistral-7B-Base2025.11 | 8.6 | — | |
| KTOBackbone=Mistral-7B-Base2025.11 | 8.2 | — | |
| TIS-DPOBackbone=Mistral-7B-Base2025.11 | 8.2 | — | |
| DPOBackbone=Mistral-7B-Base2025.11 | 8.1 | — | |
| RTOBackbone=Mistral-7B-Base2025.11 | 8.1 | — | |
| TDPOBackbone=Mistral-7B-Base2025.11 | 7.9 | — | |
| IPOBackbone=Mistral-7B-Base2025.11 | 7.5 | — | |
| DPO beta=0.01Base Model=Qwen2-1.5B-Instruct, beta=0.012025.03 | 7 | 6.8 | |
| SimPOBase Model=Qwen2-1.5B-Instruct2025.03 | 6.9 | 4.2 | |
| RTOBase Model=Qwen2-1.5B-Instruct2025.03 | 6.7 | 5.9 | |
| SimPOBase Model=Qwen2.5-1.5B-Instruct2025.03 | 4 | 4 | |
| KTOBase Model=Qwen2-1.5B-Instruct2025.03 | 3.4 | 4.2 | |
| TDPO1Base Model=Qwen2-1.5B-Instruct2025.03 | 3.2 | 3.9 | |
| DPOBase Model=Qwen2-1.5B-Instruct2025.03 | 3 | 3.6 | |
| PPOBase Model=Qwen2-1.5B-Instruct2025.03 | 2.7 | 3 | |
| Initial ModelBase Model=Qwen2-1.5B-Instruct2025.03 | 1.8 | 2.8 | |
| TDPO2Base Model=Qwen2-1.5B-Instruct2025.03 | 1.3 | 1.9 |