Reward Modeling on RewardBench (full)
99.2Chat ScoreHyRe (best weight oracle)* + Skywork-Llama-3.1-8B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| HyRe (best weight oracle)* + Skywork-Llama-3.1-8BType=Seq + Oracle, oracle=best weight2024.12 | 99.2 | 93 | 96.5 | 98.8 | 97.2 | — | |
| HyRe (best weight oracle)* + GRM-Gemma2-2BType=Seq + Oracle, oracle=best weight2024.12 | 98.3 | 83.4 | 96.7 | 94.9 | 93.1 | — | |
| HyRe (best head oracle)* + Skywork-Llama-3.1-8BType=Seq + Oracle, oracle=best head2024.12 | 98.3 | 91.2 | 95.7 | 98.4 | 96.4 | — | |
| Vanilla RMModel Scale=Gemma-2B-it, k=3.2e42026.04 | 97.77 | 54.82 | 83.24 | 66.18 | 75.5 | 72.46 | |
| Tulu-2-DPO-70BType=DPO2024.12 | 97.5 | 60.5 | 84.5 | 74.1 | 79.1 | — | |
| HyRe (best head oracle)* + GRM-Gemma2-2BType=Seq + Oracle, oracle=best head2024.12 | 97.2 | 80 | 96.2 | 94.2 | 91.8 | — | |
| CARPModel Scale=Gemma-2B-it, k=3.2e42026.04 | 96.93 | 58.99 | 79.05 | 71.56 | 76.63 | 74.54 | |
| StableLM-2-12B-ChatType=DPO2024.12 | 96.6 | 55.5 | 78.1 | 89.4 | 79.9 | — | |
| GPT-4 (May 2024)Type=Gen2024.12 | 96.6 | 70.4 | 86.5 | 84.9 | 84.6 | — | |
| INF-ORM-Llama3.1-70BType=Seq2024.12 | 96.6 | 91 | 93.6 | 99.1 | 95.1 | — | |
| LLaMA-3-Tulu-2-DPO-70BType=DPO2024.12 | 96.4 | 57.5 | 74.9 | 80.2 | 77.2 | — | |
| Claude-3 Sonnet (June 2024)Type=Gen2024.12 | 96.4 | 74 | 81.6 | 84.7 | 84.2 | — | |
| HyRe (uniform) + GRM-Gemma2-2BType=Seq, Aggregation=uniform2024.12 | 96.4 | 73.1 | 87.4 | 89.8 | 87.1 | — | |
| HyRe (N=10) + GRM-Gemma2-2BType=Seq + HyRe, N=102024.12 | 96.4 | 74.7 | 92.4 | 93.5 | 89.7 | — | |
| RRMModel Scale=Gemma-2B-it, k=3.2e42026.04 | 96.37 | 57.24 | 68.92 | 79.52 | 75.51 | 75.51 | |
| GPT-4 (Aug 2024)Type=Gen2024.12 | 96.1 | 76.1 | 88.1 | 86.6 | 86.7 | — | |
| Skywork-Reward-Gemma-2-27BType=Seq2024.12 | 96.1 | 89.9 | 93 | 98.1 | 94.3 | — | |
| HyRe (N=10) + Skywork-Llama-3.1-8BType=Seq + HyRe, N=102024.12 | 95.9 | 89.3 | 93.5 | 97.9 | 95 | — | |
| Tulu-2-DPO-13BType=DPO2024.12 | 95.8 | 58.3 | 79.5 | 73.2 | 76.7 | — | |
| HyRe (N=5) + Skywork-Llama-3.1-8BType=Seq + HyRe, N=52024.12 | 95.5 | 88.6 | 93.2 | 97.8 | 94.7 | — | |
| HyRe (N=1) + Skywork-Llama-3.1-8BType=Seq + HyRe, N=12024.12 | 95.2 | 87.8 | 93 | 97.5 | 94.3 | — | |
| Mixtral-8x7B-Instruct-v0.1Type=DPO2024.12 | 95 | 64 | 72.6 | 78.7 | 77.6 | — | |
| HyRe (N=5) + GRM-Gemma2-2BType=Seq + HyRe, N=52024.12 | 95 | 72.5 | 90.3 | 93.1 | 88.5 | — | |
| HyRe (uniform) + Skywork-Llama-3.1-8BType=Seq, Aggregation=uniform2024.12 | 95 | 87.2 | 93 | 96.8 | 94 | — | |
| Skywork-Llama-3.1-8BType=Seq2024.12 | 94.7 | 88.6 | 92.7 | 96.7 | 94 | — | |
| Gemini-1.5-Pro-0924Type=Gen2024.12 | 94.1 | 77 | 85.8 | 90.2 | 86.8 | — | |
| GRM-Gemma2-2BType=Seq2024.12 | 93 | 77.2 | 92.2 | 91.2 | 88.4 | — | |
| HyRe (N=1) + GRM-Gemma2-2BType=Seq + HyRe, N=12024.12 | 92.4 | 71.5 | 85.1 | 92.5 | 86.5 | — | |
| HyRe (best weight oracle)* + GRM-Gemma-2BType=Seq + Oracle, oracle=best weight2024.12 | 92.3 | 81.8 | 92.5 | 93.1 | 90 | — | |
| HyRe (best head oracle)* + GRM-Gemma-2BType=Seq + Oracle, oracle=best head2024.12 | 91.1 | 78.1 | 91.9 | 92.3 | 88.6 | — | |
| HyRe (N=10) + GRM-Gemma-2BType=Seq + HyRe, N=102024.12 | 90.4 | 72.5 | 90 | 92.3 | 87.2 | — | |
| HyRe (N=5) + GRM-Gemma-2BType=Seq + HyRe, N=52024.12 | 90.3 | 72.6 | 89.1 | 91.4 | 86.4 | — | |
| GRM-Gemma-2BType=Seq2024.12 | 89.4 | 75.2 | 84.5 | 88.8 | 84.5 | — | |
| HyRe (uniform) + GRM-Gemma-2BType=Seq, Aggregation=uniform2024.12 | 88.6 | 72.9 | 83.7 | 89.8 | 84.5 | — | |
| HyRe (N=1) + GRM-Gemma-2BType=Seq + HyRe, N=12024.12 | 88.5 | 72.7 | 85.5 | 91.4 | 85.3 | — | |
| Full SetSelection Strategy=Full Set2025.08 | 81.98 | 60.39 | 79.06 | 56.24 | — | 70.08 | |
| Difficulty-Based Preference Data SelectionSelection Strategy=Ours2025.08 | 80.73 | 63.42 | 80.59 | 55.31 | — | 70.56 | |
| ZIPSelection Strategy=ZIP, Adapted from=IFT-oriented data selection2025.08 | 79.33 | 57.34 | 79.26 | 57.64 | — | 70.12 | |
| RandomSelection Strategy=Random2025.08 | 78.74 | 51.55 | 76.98 | 55.92 | — | 68.82 | |
| SDPOSelection Strategy=SDPO2025.08 | 78.6 | 55.93 | 78.02 | 55.08 | — | 69.23 | |
| DiverseEvolSelection Strategy=DiverseEvol, Adapted from=IFT-oriented data selection2025.08 | 77.91 | 53.64 | 78.64 | 56.31 | — | 69.54 |