Pairwise evaluation on BIGGEN
78.33Human AgreementHuman-crafted (existing) Rubrics
Evaluation Results
| Method | Links | |
|---|---|---|
| Human-crafted (existing) RubricsJudge Model=Claude Sonnet 4, Rubric Granularity=Instance-specific2026.05 | 78.33 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=22026.05 | 76.96 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=22026.05 | 76.96 | |
| Human-crafted (existing) RubricsJudge Model=Qwen3 14B, Rubric Granularity=Instance-specific2026.05 | 76.68 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=12026.05 | 76.28 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=12026.05 | 76.28 | |
| Human-crafted (existing) RubricsJudge Model=Llama 3.1 70B, Rubric Granularity=Instance-specific2026.05 | 76.15 | |
| Training-free Rubric GenerationJudge Model=Claude Sonnet 4, Rubric Granularity=Instance-specific2026.05 | 74.89 | |
| Highest Training-free baselineJudge=Claude Sonnet 42026.05 | 74.89 | |
| Highest Training-freeJudge=Claude Sonnet 42026.05 | 74.89 | |
| DnA-EvalJudge Model=Claude Sonnet 4, Approach Type=Training-free Approach2026.05 | 74.82 | |
| Training-free Rubric GenerationJudge Model=Claude Sonnet 4, Rubric Granularity=Dataset-specific2026.05 | 74.75 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=12026.05 | 74.67 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=12026.05 | 74.67 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=22026.05 | 74.46 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=22026.05 | 74.46 | |
| Human-crafted (existing) RubricsJudge Model=Claude Sonnet 4, Rubric Granularity=Dataset-specific2026.05 | 74.36 | |
| Training-free Rubric GenerationJudge Model=Qwen3 14B, Rubric Granularity=Instance-specific2026.05 | 72.96 | |
| Highest Training-free baselineJudge=Qwen3 14B2026.05 | 72.96 | |
| Highest Training-freeJudge=Qwen3 14B2026.05 | 72.96 | |
| Human-crafted (existing) RubricsJudge Model=Qwen3 14B, Rubric Granularity=Dataset-specific2026.05 | 72.53 | |
| Training-free Rubric GenerationJudge Model=Qwen3 14B, Rubric Granularity=Dataset-specific2026.05 | 72.42 | |
| RubricHubJudge Model=Claude Sonnet 4, Approach Type=Training-free Approach2026.05 | 72.22 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=22026.05 | 71.92 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=22026.05 | 71.92 | |
| Training-free Rubric GenerationJudge Model=Llama 3.1 70B, Rubric Granularity=Instance-specific2026.05 | 71.74 | |
| Highest Training-free baselineJudge=Llama 3.1 70B2026.05 | 71.74 | |
| Highest Training-freeJudge=Llama 3.1 70B2026.05 | 71.74 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=12026.05 | 71.1 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=12026.05 | 71.1 | |
| DnA-EvalJudge Model=Qwen3 14B, Approach Type=Training-free Approach2026.05 | 71.01 | |
| DnA-EvalJudge Model=Llama 3.1 70B, Approach Type=Training-free Approach2026.05 | 70.34 | |
| Human-crafted (existing) RubricsJudge Model=Llama 3.1 70B, Rubric Granularity=Dataset-specific2026.05 | 70.2 | |
| Prometheus 8x7BApproach Type=Fine-tuned Model, Rubric Granularity=Instance-specific2026.05 | 69.73 | |
| Training-free Rubric GenerationJudge Model=Llama 3.1 70B, Rubric Granularity=Dataset-specific2026.05 | 69.73 | |
| RubricHubJudge Model=Llama 3.1 70B, Approach Type=Training-free Approach2026.05 | 69.68 | |
| CheckEvalJudge Model=Claude Sonnet 4, Approach Type=Training-free Approach2026.05 | 68.82 | |
| Prometheus 8x7BApproach Type=Fine-tuned Model, Rubric Granularity=Dataset-specific2026.05 | 68.3 | |
| RubricHubJudge Model=Qwen3 14B, Approach Type=Training-free Approach2026.05 | 68.28 | |
| CheckEvalJudge Model=Qwen3 14B, Approach Type=Training-free Approach2026.05 | 65.01 | |
| CheckEvalJudge Model=Llama 3.1 70B, Approach Type=Training-free Approach2026.05 | 64.29 |