Rebuttal Quality Evaluation on R2 (test)
9.39Rigor (C)RebuttalAgent
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RebuttalAgentReward model=RebuttalRM2026.01 | 9.39 | 9.35 | 9.51 | 9.4 | 9.32 | 9.29 | 9.53 | 8.95 | 9.7 | 9.61 | 9.45 | 9.89 | 9.45 | |
| RebuttalAgentReward model=GPT-4.1-based2026.01 | 9.33 | 9.24 | 8.85 | 9.32 | 9.16 | 9.82 | 9.35 | 9.07 | 9.3 | 9.24 | 9.38 | 9.18 | 9.35 | |
| RebuttalAgentReward component=w/o RAnalysis2026.01 | 9.25 | 9.23 | 9.79 | 9.2 | 9.18 | 9.39 | 9 | 8.87 | 9.27 | 9.59 | 9.41 | 9.45 | 9.23 | |
| RebuttalAgent2026.01 | 9.23 | 8.91 | 9.59 | 9.18 | 8.95 | 9.37 | 9.09 | 8.54 | 9.65 | 9.43 | 9.2 | 9.5 | 9.42 | |
| RebuttalAgentReward model=Llama-based2026.01 | 9.23 | 9.1 | 9.16 | 9.29 | 9.11 | 9.24 | 9.16 | 8.67 | 9.05 | 9.57 | 9.35 | 9.39 | 9.2 | |
| RebuttalAgentScoring Model=GPT-4.12026.01 | 9.18 | 8.66 | 9.95 | 9.13 | 8.67 | 9.87 | 9.12 | 8.38 | 9.81 | 9.3 | 8.82 | 9.9 | 9.27 | |
| RebuttalAgentReward weights=w/ Distinct weights2026.01 | 9.08 | 8.54 | 9.53 | 9.04 | 8.63 | 9.23 | 9.05 | 8.32 | 9.85 | 9.34 | 9.08 | 9.38 | 9.27 | |
| RebuttalAgentData Ablation=w/o Thinking2026.01 | 9.06 | 9 | 9.18 | 9.02 | 8.92 | 9.13 | 8.96 | 8.6 | 9.2 | 9.35 | 9.16 | 9.55 | 9.37 | |
| RebuttalAgentReward component=w/o RFormat2026.01 | 9.06 | 8.91 | 9.22 | 9.04 | 8.74 | 9.3 | 8.88 | 8.29 | 9.67 | 9.37 | 9.14 | 9.35 | 9.32 | |
| o3Scoring Model=GPT-4.12026.01 | 9.02 | 8.76 | 9.7 | 8.86 | 8.61 | 9.4 | 8.65 | 8.23 | 9.15 | 9.23 | 8.81 | 9.58 | 9.1 | |
| RebuttalAgentData Ablation=w/o Strategy2026.01 | 9.01 | 8.89 | 9.93 | 9 | 8.85 | 9.3 | 8.88 | 8.49 | 9.82 | 9.27 | 9.06 | 9.33 | 9.31 | |
| o32026.01 | 9 | 8.99 | 9.55 | 8.84 | 8.78 | 9.45 | 8.58 | 8.43 | 9.22 | 9.34 | 9.12 | 9.5 | 9.21 | |
| RebuttalAgentData Ablation=w/o ToM2026.01 | 8.91 | 8.21 | 9.29 | 8.88 | 8.3 | 9.28 | 8.7 | 7.87 | 9.38 | 9.22 | 8.86 | 9.58 | 9.04 | |
| TSRbase_model=o32026.01 | 8.89 | 9.1 | 9.68 | 8.95 | 8.91 | 9.28 | 8.69 | 8.56 | 9.45 | 9.18 | 9.35 | 9.45 | 9.34 | |
| RebuttalAgentReward model=Qwen3-4B-based2026.01 | 8.79 | 8.54 | 9.73 | 8.6 | 8.24 | 9.44 | 8.32 | 7.84 | 9.17 | 9.12 | 8.76 | 9.72 | 8.98 | |
| o3Evaluation Judge=GPT-52026.01 | 8.75 | 8.13 | 9.15 | 8.57 | 7.98 | 8.97 | 8.51 | 7.63 | 8.78 | 8.86 | 8.32 | 9.18 | 8.64 | |
| RebuttalAgentTraining Ablation=RL-only2026.01 | 8.63 | 8.27 | 9.42 | 8.47 | 8.07 | 9.01 | 8.21 | 7.56 | 8.34 | 9.05 | 8.71 | 9.61 | 8.79 | |
| DeepSeek-R1Scoring Model=GPT-4.12026.01 | 8.61 | 8.09 | 9.09 | 8.56 | 8.11 | 8.87 | 8.34 | 7.73 | 8.63 | 9 | 8.54 | 9.53 | 8.7 | |
| Self-Refined2026.01 | 8.55 | 8.08 | 9.04 | 8.47 | 8.04 | 8.88 | 8.19 | 7.56 | 8.52 | 9.08 | 8.75 | 9.59 | 8.72 | |
| RebuttalAgentReward component=w/o RResponse2026.01 | 8.51 | 7.9 | 9.02 | 8.41 | 7.91 | 8.63 | 8.17 | 7.51 | 8.25 | 9.05 | 8.68 | 9.61 | 8.63 | |
| DeepSeek-V3Scoring Model=GPT-4.12026.01 | 8.48 | 7.75 | 8.82 | 8.45 | 7.79 | 8.72 | 8.24 | 7.43 | 8.55 | 8.86 | 8.33 | 9.32 | 8.5 | |
| DeepSeek-R12026.01 | 8.47 | 7.9 | 8.9 | 8.46 | 8.03 | 8.75 | 8.29 | 7.71 | 8.6 | 9.03 | 8.7 | 9.54 | 8.64 | |
| TSRbase_model=GPT-4.12026.01 | 8.47 | 7.63 | 8.53 | 8.12 | 7.94 | 8.85 | 7.9 | 7.51 | 8.45 | 9.07 | 8.42 | 9.16 | 8.76 | |
| RebuttalAgentTraining Ablation=w DPO2026.01 | 8.47 | 8.13 | 9.36 | 8.32 | 7.92 | 9 | 8.11 | 7.57 | 8.82 | 8.94 | 8.55 | 9.46 | 8.68 | |
| Deepseek-V32026.01 | 8.43 | 7.67 | 8.83 | 8.42 | 7.71 | 8.72 | 8.18 | 7.35 | 8.59 | 8.94 | 8.45 | 9.41 | 8.51 | |
| GPT-4.1Scoring Model=GPT-4.12026.01 | 8.36 | 7.83 | 8.76 | 8.3 | 7.77 | 8.51 | 8.1 | 7.39 | 8.23 | 8.8 | 8.34 | 9.25 | 8.43 | |
| GPT-4.12026.01 | 8.34 | 7.86 | 8.8 | 8.27 | 7.79 | 8.62 | 8.05 | 7.28 | 8.2 | 8.91 | 8.57 | 9.42 | 8.5 | |
| Strategy-Prompt2026.01 | 8.26 | 7.41 | 8.32 | 8.33 | 7.77 | 8.51 | 8.13 | 7.41 | 7.95 | 8.85 | 8.44 | 9.46 | 8.37 | |
| RebuttalAgentEvaluation Judge=GPT-52026.01 | 8.24 | 6.31 | 8.7 | 8.04 | 6.33 | 8.38 | 8.354 | 6.09 | 8.12 | 8.37 | 7.33 | 8.75 | 7.83 | |
| RebuttalAgentTraining Ablation=SFT-only2026.01 | 8.2 | 7.6 | 8.42 | 8.17 | 7.6 | 8.28 | 8.02 | 7.31 | 7.84 | 8.76 | 8.34 | 9.16 | 8.27 | |
| GLM-4-9B2026.01 | 8.08 | 7.46 | 8.69 | 7.97 | 7.24 | 8.26 | 7.84 | 6.9 | 8.11 | 8.52 | 8.02 | 8.99 | 8.13 | |
| DeepSeek-R1Evaluation Judge=GPT-52026.01 | 8.03 | 6.27 | 7.36 | 7.82 | 6.34 | 7.09 | 7.88 | 6.03 | 6.9 | 8.38 | 7.47 | 8.45 | 7.46 | |
| Qwen3-8BScoring Model=GPT-4.12026.01 | 7.99 | 7.3 | 8.11 | 7.84 | 7.09 | 7.61 | 7.7 | 6.75 | 7.28 | 8.5 | 7.92 | 8.77 | 7.9 | |
| Qwen3-8B2026.01 | 7.96 | 7.33 | 8.18 | 7.84 | 7.11 | 7.76 | 7.68 | 6.73 | 7.39 | 8.51 | 8.08 | 8.87 | 7.96 | |
| Gemini-2.5Scoring Model=GPT-4.12026.01 | 7.94 | 6.99 | 6.82 | 8.09 | 7.41 | 7.3 | 8.03 | 7.12 | 7.08 | 8.58 | 8.05 | 8.84 | 7.79 | |
| Gemini-2.52026.01 | 7.89 | 6.91 | 6.63 | 8.06 | 7.41 | 7.26 | 7.87 | 7.09 | 6.89 | 8.56 | 8.11 | 8.83 | 7.75 | |
| Qwen3-4B2026.01 | 7.84 | 7.05 | 7.42 | 7.77 | 6.98 | 6.99 | 7.72 | 6.69 | 6.83 | 8.48 | 8.02 | 8.66 | 7.69 | |
| Llama-3.1-8B2026.01 | 7.77 | 6.69 | 7.32 | 7.71 | 6.76 | 7.02 | 7.54 | 6.3 | 6.49 | 8.12 | 7.42 | 8.25 | 7.44 | |
| DeepSeek-V3Evaluation Judge=GPT-52026.01 | 7.74 | 5.62 | 6.96 | 7.58 | 5.71 | 6.83 | 7.73 | 5.45 | 6.75 | 8.13 | 6.99 | 8.06 | 7.07 | |
| GPT-4.1Evaluation Judge=GPT-52026.01 | 7.63 | 5.93 | 7.3 | 7.47 | 5.91 | 6.9 | 7.36 | 5.45 | 6.51 | 8.09 | 7.17 | 8.2 | 7.72 | |
| Gemini-2.5Evaluation Judge=GPT-52026.01 | 7.26 | 4.77 | 4.53 | 7.31 | 5.32 | 5.01 | 7.11 | 4.89 | 4.16 | 7.87 | 6.64 | 7.27 | 6.21 | |
| RebuttalFTbase_model=Qwen3-8B2026.01 | 6.91 | 6.07 | 6.8 | 6.58 | 5.72 | 6.24 | 6.52 | 5.5 | 5.94 | 6.55 | 5.79 | 6.63 | 6.35 | |
| Qwen3-8BEvaluation Judge=GPT-52026.01 | 6.76 | 4.74 | 6.19 | 6.38 | 4.41 | 5.3 | 6.53 | 4.24 | 4.92 | 7.32 | 6.07 | 7.14 | 6.02 | |
| RebuttalFTScoring Model=GPT-4.12026.01 | 6.74 | 5.75 | 6.08 | 6.35 | 5.36 | 5.32 | 6.71 | 5.58 | 5.42 | 6.01 | 5.07 | 5.55 | 5.8 | |
| RebuttalFTEvaluation Judge=GPT-52026.01 | 5.65 | 3.65 | 4.61 | 4.91 | 3.14 | 3.57 | 5.42 | 3.14 | 3.14 | 4.89 | 3.45 | 4.39 | 4.22 |