Reward Modeling on RewardBench unified-feedback (test)
84Average ScoreStarling-RM-34B
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Starling-RM-34BTraining Data Size=400K2024.06 | 84 | 96.9 | 59 | 89.9 | 90.3 | |
| GRM (linear) w/ sftBase Model=Mistral 7b Instruct, Training Data Size=400K, Head=Linear, Regularization=sft2024.06 | 79.5 | 97.8 | 54.6 | 86.3 | 79.2 | |
| GRM (linear) w/ dpo norefBase Model=Mistral 7b Instruct, Training Data Size=400K, Head=Linear, Regularization=dpo noref2024.06 | 78.3 | 98 | 53.3 | 86.4 | 75.3 | |
| GRM w/ dpo norefBase Model=Mistral 7b Instruct, Training Data Size=400K, Regularization=dpo noref2024.06 | 78 | 97.8 | 54 | 85.7 | 74.4 | |
| GRM w/ sftBase Model=Mistral 7b Instruct, Training Data Size=400K, Regularization=sft2024.06 | 77.6 | 98 | 55.3 | 85.8 | 71.2 | |
| Classifier + EnsembleBase Model=Mistral 7b Instruct, Training Data Size=400K2024.06 | 76.6 | 96.6 | 51.8 | 85.1 | 73 | |
| Classifier (baseline)Base Model=Mistral 7b Instruct, Training Data Size=400K2024.06 | 76.3 | 96.6 | 52.4 | 86.7 | 69.5 | |
| Classifier + label smoothBase Model=Mistral 7b Instruct, Training Data Size=400K2024.06 | 76.3 | 97.2 | 49.8 | 85.8 | 72.3 | |
| Starling-RM-7BTraining Data Size=400K2024.06 | 76.2 | 98 | 43.4 | 88.6 | 74.6 | |
| Classifier + marginBase Model=Mistral 7b Instruct, Training Data Size=400K2024.06 | 74.5 | 96.4 | 51.5 | 85.3 | 64.8 | |
| GRM (linear) w/ sftBase Model=Gemma 2b it, Training Data Size=400K, Head=Linear, Regularization=sft2024.06 | 71.5 | 96.1 | 40.1 | 80.3 | 69.3 | |
| Classifier + EnsembleBase Model=Gemma 2b it, Training Data Size=400K2024.06 | 71 | 98 | 37.5 | 77.3 | 71.3 | |
| GRM w/ sftBase Model=Gemma 2b it, Training Data Size=400K, Regularization=sft2024.06 | 70.8 | 97.8 | 42.1 | 77.9 | 65.2 | |
| Classifier + label smoothBase Model=Gemma 2b it, Training Data Size=400K2024.06 | 70.6 | 94.4 | 37.3 | 73.2 | 77.4 | |
| Classifier + marginBase Model=Gemma 2b it, Training Data Size=400K2024.06 | 70.2 | 95.8 | 38.4 | 73.9 | 72.5 | |
| GRM (linear) w/ dpo norefBase Model=Gemma 2b it, Training Data Size=400K, Head=Linear, Regularization=dpo noref2024.06 | 70.2 | 96.7 | 39 | 76.4 | 68.5 | |
| GRM w/ dpo norefBase Model=Gemma 2b it, Training Data Size=400K, Regularization=dpo noref2024.06 | 70.2 | 95.8 | 40.1 | 78.7 | 66.2 | |
| UltraRM-13BTraining Data Size=400K2024.06 | 69.8 | 96.1 | 55.3 | 45.8 | 82 | |
| Classifier (baseline)Base Model=Gemma 2b it, Training Data Size=400K2024.06 | 68.2 | 95.5 | 38 | 73.8 | 65.3 | |
| PairRMTraining Data Size=400K2024.06 | 58.7 | 90.2 | 53 | 31.5 | 60 |