Pairwise evaluation on AlpacaEval
72.4Human AgreementFine-tuned Rubric Generator
Evaluation results
| Method | Links | |
|---|---|---|
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=12026.05 | 72.4 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=12026.05 | 72.4 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=22026.05 | 72.34 | |
| Fine-tuned Rubric GeneratorJudge=Qwen3 14B, Iteration=22026.05 | 72.34 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=22026.05 | 72.22 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=22026.05 | 72.22 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=12026.05 | 72.09 | |
| Fine-tuned Rubric GeneratorJudge=Claude Sonnet 4, Iteration=12026.05 | 72.09 | |
| Training-free Rubric GenerationJudge Model=Qwen3 14B, Rubric Granularity=Instance-specific2026.05 | 71.96 | |
| Highest Training-free baselineJudge=Qwen3 14B2026.05 | 71.96 | |
| Highest Training-freeJudge=Qwen3 14B2026.05 | 71.96 | |
| Training-free Rubric GenerationJudge Model=Claude Sonnet 4, Rubric Granularity=Instance-specific2026.05 | 71.65 | |
| Highest Training-free baselineJudge=Claude Sonnet 42026.05 | 71.65 | |
| Highest Training-freeJudge=Claude Sonnet 42026.05 | 71.65 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=22026.05 | 70.72 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=22026.05 | 70.72 | |
| DnA-EvalJudge Model=Claude Sonnet 4, Approach Type=Training-free Approach2026.05 | 70.56 | |
| Human-crafted (existing) RubricsJudge Model=Claude Sonnet 4, Rubric Granularity=Dataset-specific2026.05 | 70.4 | |
| Human-crafted (existing) RubricsJudge Model=Qwen3 14B, Rubric Granularity=Dataset-specific2026.05 | 70.09 | |
| DnA-EvalJudge Model=Qwen3 14B, Approach Type=Training-free Approach2026.05 | 69.78 | |
| Training-free Rubric GenerationJudge Model=Llama 3.1 70B, Rubric Granularity=Instance-specific2026.05 | 69.78 | |
| Training-free Rubric GenerationJudge Model=Claude Sonnet 4, Rubric Granularity=Dataset-specific2026.05 | 69.78 | |
| Highest Training-free baselineJudge=Llama 3.1 70B2026.05 | 69.78 | |
| Highest Training-freeJudge=Llama 3.1 70B2026.05 | 69.78 | |
| Training-free Rubric GenerationJudge Model=Qwen3 14B, Rubric Granularity=Dataset-specific2026.05 | 69.47 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=12026.05 | 69.47 | |
| Fine-tuned Rubric GeneratorJudge=Llama 3.1 70B, Iteration=12026.05 | 69.47 | |
| Prometheus 8x7BApproach Type=Fine-tuned Model, Rubric Granularity=Dataset-specific2026.05 | 68.54 | |
| Human-crafted (existing) RubricsJudge Model=Llama 3.1 70B, Rubric Granularity=Dataset-specific2026.05 | 68.22 | |
| Training-free Rubric GenerationJudge Model=Llama 3.1 70B, Rubric Granularity=Dataset-specific2026.05 | 67.91 | |
| CheckEvalJudge Model=Claude Sonnet 4, Approach Type=Training-free Approach2026.05 | 66.82 | |
| RubricHubJudge Model=Claude Sonnet 4, Approach Type=Training-free Approach2026.05 | 64.64 | |
| RubricHubJudge Model=Llama 3.1 70B, Approach Type=Training-free Approach2026.05 | 62.93 | |
| RubricHubJudge Model=Qwen3 14B, Approach Type=Training-free Approach2026.05 | 62.93 | |
| DnA-EvalJudge Model=Llama 3.1 70B, Approach Type=Training-free Approach2026.05 | 62.46 | |
| CheckEvalJudge Model=Qwen3 14B, Approach Type=Training-free Approach2026.05 | 60.44 | |
| CheckEvalJudge Model=Llama 3.1 70B, Approach Type=Training-free Approach2026.05 | 58.72 |