ResearchBenchmarksPolicy Optimization on RLHFFollow8.3True ScoreORPO2.07043.68775.3056.9223Apr 13, 2026Evaluation ResultsMethodMethodLinksTrue ScoreProxy ScoreWorst Case ValueORPOEnvironment=RLHFEnvironment=RLHF2026.048.30.63-1.84ORPO2026.048.30.63-1.84Max-MinEnvironment=RLHFEnvironment=RLHF2026.045.380.84-0.1Max-Min2026.045.380.84-0.1EnsembleEnvironment=RLHFEnvironment=RLHF2026.042.311.26-1.7