UltraFeedback
Benchmarks
TaskDatasetSOTA resultTrendResults
Ultrafeedback 40% flipping ratio
78.87Accuracy
12
Ultrafeedback 20% flipping ratio
78.8Accuracy
12
UltraFeedback (test)
74.18Accuracy
11
UltraFeedback
69.92Accuracy
11
Ultrafeedback
73.66Accuracy
9
UltraFeedback Property Constraints Satisfaction (test)
0.07Worst-case Size Distortion (Helpfulness & Instruction-following)
9
UltraFeedback kappa_max = 10 (Aggregate)
0.847Q-hat
8
UltraFeedback (test_prefs)
57.65Pair Accuracy
8
UltraFeedback Cleaned
92.36Total Score
8
UltraFeedback (in-domain)
80.6Win Rate (KL, alpha=1)
8
UltraFeedback 500 held-out users (test)
70.53Test Accuracy
7
UltraFeedback (test)
18.8Win Rate
6
UltraFeedback pool (test)
51.8Instruction Following Win Rate
6
UltraFeedback
23Instruction Following Win (%)
6
UltraFeedback (test)
62.35Pairwise Win Rate (excl. ties)
5
UltraFeedback (test)
-3.12Score
5
UltraFeedback 200K samples (test)
81.9Win Rate
5
UltraFeedback
0.623MAE
5
UltraFeedback 50 sampled questions
62Win Rate (Expert 1)
5
UltraFeedback Near-tie 0.4 noisy (test)
0.37Kendall’s Tau
4
UltraFeedback Top-rank 0.4 noisy (test)
0.356Kendall's Tau
4
UltraFeedback Clean (test)
0.39Kendall's τ
4
UltraFeedback core250 (test)
1.103Reward Score Difference (TEA vs GRPO)
4
UltraFeedback core500 (held-out)
0.467bo1 Score
4
ultrafeedback-prompt (test)
52Win Rate
4