Preference Prediction on Composite Dataset (Arena-Expert-5K, HelpSteer3, HH-RLHF, UltraFeedback)
70.5AccuracyPREMISE (+ pref repair)
Evaluation results
| Method | Links | |
|---|---|---|
| PREMISE (+ pref repair)LLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 70.5 | |
| PREMISE (+ pref repair)LLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 70.5 | |
| Auto-RubricLLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 70.5 | |
| Auto-RubricLLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 70.3 | |
| AgentEvalLLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 70.2 | |
| Inverse CAILLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 69.9 | |
| CritiQLLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 69.9 | |
| AutoRuleLLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 69.8 | |
| AgentEvalLLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 69.7 | |
| AutoRuleLLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 69.6 | |
| CritiQLLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 69.5 | |
| PREMISE (+ pref repair)LLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 69.2 | |
| AutoRuleLLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 69 | |
| CritiQLLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 69 | |
| Auto-RubricLLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 68.8 | |
| PREMISE (+ pref repair)LLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 68.6 | |
| AutoRuleLLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 68.4 | |
| Inverse CAILLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 68.3 | |
| AgentEvalLLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 68.2 | |
| Auto-RubricLLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 68.1 | |
| AgentEvalLLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 68.1 | |
| Auto-RubricLLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 68 | |
| PREMISE (discovered)LLM Judge=Kimi, Prompt Template=direct-verdict2026.05 | 67.9 | |
| CritiQLLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 67.9 | |
| CritiQLLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 67.5 | |
| Inverse CAILLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 67.4 | |
| AutoRuleLLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 67.4 | |
| AgentEvalLLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 67.1 | |
| PREMISE (+ pref repair)LLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 67 | |
| Inverse CAILLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 67 | |
| Auto-RubricLLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 66.4 | |
| Inverse CAILLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 66.4 | |
| Inverse CAILLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 66.2 | |
| AgentEvalLLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 66.1 | |
| PREMISE (discovered)LLM Judge=Kimi, Prompt Template=per-rubric2026.05 | 66 | |
| PREMISE (+ pref repair)LLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 65.8 | |
| AutoRuleLLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 65.3 | |
| PREMISE (discovered)LLM Judge=DS-V3, Prompt Template=direct-verdict2026.05 | 65.2 | |
| CritiQLLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 64.8 | |
| PREMISE (discovered)LLM Judge=DS-V3, Prompt Template=per-rubric2026.05 | 64 | |
| PREMISE (discovered)LLM Judge=Qwen-32B, Prompt Template=direct-verdict2026.05 | 63.6 | |
| PREMISE (discovered)LLM Judge=Qwen-32B, Prompt Template=per-rubric2026.05 | 63 |