LLM-as-a-Judge Calibration on RewardBench (test)
0.022Test Risk (MSE)Test oracle envelope
Evaluation results
| Method | Links | |
|---|---|---|
| Test oracle envelopeCalibration Budget=Largest available, Selection Strategy=Test risk oracle, Deployable=false2026.05 | 0.022 | |
| Information-first + val. KCalibration Budget=Largest available, Selection Strategy=Information-first, Prefix Selection=Validation K2026.05 | 0.024 | |
| Complexity-penalized + val. KCalibration Budget=Largest available, Selection Strategy=Complexity-penalized, Prefix Selection=Validation K2026.05 | 0.025 | |
| Path-family envelopeCalibration Budget=Largest available, Selection Strategy=Path-family envelope, Optimization=Validation risk2026.05 | 0.025 | |
| Random path + val. KCalibration Budget=Largest available, Selection Strategy=Random path, Prefix Selection=Validation K2026.05 | 0.027 | |
| All seven judgesCalibration Budget=Largest available, Selection Strategy=All judges aggregation2026.05 | 0.028 | |
| Best single judge (val.)Calibration Budget=Largest available, Selection Strategy=Best single judge (validation)2026.05 | 0.032 |