Three-objective alignment on Helpful Assistant (test)
0.74Helpfulness ScoreRLHF-r1
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RLHF-r1SFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.74 | -0.41 | -0.22 | |
| RLHF-r1SFT model=OpenChat-v3.52025.05 | 0.72 | -0.38 | -0.2 | |
| RLHF-r1SFT model=Llama-3.1-8B2025.05 | 0.69 | -0.33 | -0.18 | |
| RLHF-r1SFT model=phi-1.52025.05 | 0.65 | -0.73 | -0.47 | |
| PARMSFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.45 | 0.16 | 0.23 | |
| MOPO-LagSFT model=OpenChat-v3.52025.05 | 0.4 | 0.23 | 0.18 | |
| MOPO-LagSFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.39 | 0.22 | 0.17 | |
| MOPO-LagSFT model=Llama-3.1-8B2025.05 | 0.38 | 0.21 | 0.16 | |
| PARMSFT model=OpenChat-v3.52025.05 | 0.32 | 0.18 | 0.24 | |
| MOPO-LBSFT model=Llama-3.1-8B2025.05 | 0.31 | 0.2 | 0.19 | |
| PARMSFT model=Llama-3.1-8B2025.05 | 0.3 | 0.17 | 0.22 | |
| MOPO-LBSFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.3 | 0.19 | 0.18 | |
| MOPO-LBSFT model=OpenChat-v3.52025.05 | 0.29 | 0.18 | 0.17 | |
| RiCSFT model=OpenChat-v3.52025.05 | 0.27 | 0.16 | 0.12 | |
| RiCSFT model=Llama-3.1-8B2025.05 | 0.26 | 0.14 | 0.11 | |
| RiCSFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.25 | 0.15 | 0.1 | |
| MOPO-LBSFT model=phi-1.52025.05 | 0.24 | 0.13 | 0.14 | |
| PARMSFT model=phi-1.52025.05 | 0.23 | 0.12 | 0.11 | |
| MOPO-LagSFT model=phi-1.52025.05 | 0.2 | 0.1 | 0.09 | |
| DPO on DJSFT model=Llama-3.1-8B2025.05 | 0.19 | 0.1 | 0.12 | |
| RiCSFT model=phi-1.52025.05 | 0.18 | 0.1 | 0.05 | |
| DPO on DJSFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.18 | 0.09 | 0.11 | |
| DPO on DJSFT model=OpenChat-v3.52025.05 | 0.17 | 0.08 | 0.1 | |
| DPO on DJSFT model=phi-1.52025.05 | 0.13 | 0.05 | 0.08 | |
| MODPOSFT model=OpenChat-v3.52025.05 | 0.05 | -0.08 | 0.09 | |
| MODPOSFT model=Mistral-7b-v0.2 (Instruct)2025.05 | 0.04 | -0.09 | 0.08 | |
| MODPOSFT model=Llama-3.1-8B2025.05 | 0.03 | -0.1 | 0.07 | |
| MODPOSFT model=phi-1.52025.05 | -0.04 | -0.17 | 0 | |
| RLHF-r3SFT model=OpenChat-v3.52025.05 | -0.77 | -0.89 | 0.4 | |
| RLHF-r2SFT model=OpenChat-v3.52025.05 | -0.78 | 0.49 | -0.37 | |
| RLHF-r3SFT model=Mistral-7b-v0.2 (Instruct)2025.05 | -0.78 | -0.91 | 0.41 | |
| RLHF-r2SFT model=Mistral-7b-v0.2 (Instruct)2025.05 | -0.79 | 0.52 | -0.38 | |
| RLHF-r2SFT model=Llama-3.1-8B2025.05 | -0.8 | 0.5 | -0.39 | |
| RLHF-r3SFT model=Llama-3.1-8B2025.05 | -0.8 | -0.9 | 0.43 | |
| RLHF-r3SFT model=phi-1.52025.05 | -0.88 | -0.97 | 0.29 | |
| RLHF-r2SFT model=phi-1.52025.05 | -0.93 | 0.44 | -0.53 |