Preference-based Alignment on Stanford SHP (test)
83.5Win Rate (Llama)DPO
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DPOModel=Hermes3-8B2026.04 | 83.5 | 79.1 | 65.8 | -2.153 | 81 | 64 | |
| Pref-CTRL (Margin + Regularizer)Model=Hermes3-8B2026.04 | 80.4 | 76.4 | 61.4 | -2.166 | 81 | 65 | |
| Pref-CTRL (Margin)Model=Hermes3-8B2026.04 | 80 | 77.5 | 59.9 | -2.377 | 81 | 64 | |
| RE-ControlModel=Hermes3-8B2026.04 | 79.8 | 74.8 | 60.9 | -2.303 | 80 | 65 | |
| Pref-CTRL (Margin + Regularizer)Model=Vicuna-7B2026.04 | 73.5 | 70 | 53.7 | -2.454 | 76 | 66 | |
| Pref-CTRL (Margin)Model=Vicuna-7B2026.04 | 72.2 | 67.6 | 50.2 | -2.612 | 77 | 65 | |
| DPOModel=Vicuna-7B2026.04 | 71.8 | 70.7 | 56.3 | -2.342 | 70 | 65 | |
| Pref-CTRL (Regularizer)Model=Vicuna-7B2026.04 | 68.07 | 64.56 | 51.7 | -2.884 | 71 | 64 | |
| RE-ControlModel=Vicuna-7B2026.04 | 66.8 | 66.7 | 53.5 | -2.652 | 76 | 65 |