Alignment on UltraFeedback (test)
63.72Honesty ScoreMetaAligner-7B
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| MetaAligner-7BPolicy Model=LLaMA2-Chat-70B, GPU Hours=242.682024.03 | 63.72 | 67.05 | 70.1 | 54.7 | 63.89 | |
| Self-RefinementPolicy Model=LLaMA2-Chat-70B2024.03 | 62.91 | 49.95 | 60.77 | 57.6 | 55.05 | |
| MetaAligner-13BPolicy Model=LLaMA2-Chat-7B, GPU Hours=403.442024.03 | 59.5 | 68.5 | 64 | 55 | 61.75 | |
| MetaAligner-7BPolicy Model=LLaMA2-Chat-7B, GPU Hours=242.682024.03 | 48 | 51.33 | 55.67 | 44.33 | 49.83 | |
| MetaAligner-7B (w/o Warm Up)Policy Model=LLaMA2-Chat-7B, GPU Hours=242.682024.03 | 47.4 | 49.96 | 55.73 | 44.18 | 49.32 | |
| MetaAligner-7B (w/o Equal Pref.)Policy Model=LLaMA2-Chat-7B, GPU Hours=242.682024.03 | 43.6 | 46.8 | 53.17 | 41.7 | 46.32 | |
| MODPOPolicy Model=LLaMA2-Chat-7B, GPU Hours=405.92024.03 | 43.4 | 30.82 | 37.19 | 25 | 34.1 | |
| MAH-DPOBackbone=Llama-3.1-8B-Instruct, Configuration=Ensemble2025.10 | 36.87 | — | 24.78 | 63.89 | — | |
| SFTPolicy Model=LLaMA2-Chat-7B, GPU Hours=247.742024.03 | 36.5 | 27 | 26 | 36.5 | 31.5 | |
| Aligner-7BPolicy Model=LLaMA2-Chat-7B, GPU Hours=236.82024.03 | 36.5 | 27 | 26 | 36.5 | 31.5 | |
| MAH-DPOBackbone=Llama-3.1-8B-Instruct, Configuration=Honest Head2025.10 | 35.16 | — | 23.03 | 62.57 | — | |
| MODPOBackbone=Llama-3.1-8B-Instruct2025.10 | 34.77 | — | 23.25 | 61.75 | — | |
| MAH-DPOBackbone=Llama-3.1-8B-Instruct, Configuration=Help Head2025.10 | 34.65 | — | 22.39 | 63.09 | — | |
| MAH-DPOBackbone=Llama-3.1-8B-Instruct, Configuration=Truth Head2025.10 | 34.61 | — | 22.86 | 62.57 | — | |
| MORLHFPolicy Model=LLaMA2-Chat-7B, GPU Hours=1892.32024.03 | 33.7 | 32.18 | 26.1 | 33.7 | 31.42 | |
| DPO SoupBackbone=Llama-3.1-8B-Instruct2025.10 | 32.17 | — | 21.53 | 61.28 | — | |
| Single-Head DPOBackbone=Llama-3.1-8B-Instruct2025.10 | 30.55 | — | 20.14 | 60.43 | — | |
| BaseBackbone=Llama-3.1-8B-Instruct2025.10 | 30.42 | — | 18.88 | 58 | — | |
| SFTBackbone=Llama-3.1-8B-Instruct2025.10 | 29.98 | — | 19.92 | 55.46 | — | |
| MetaAligner-1.1BPolicy Model=LLaMA2-Chat-7B, GPU Hours=120.482024.03 | 27 | 27.67 | 33 | 25.33 | 28.25 |