Error Attribution on Aegis-Bench
16.62Pair µF1Qwen2.5-14B-Instruct + Aegis-SFT
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-14B-Instruct + Aegis-SFTModel Scale=Medium-Scale, Training Protocol=SFT2025.09 | 16.62 | 9.99 | 76.53 | 47.97 | 27.53 | 27.66 | 26.51 | |
| VerifyMAS-8B-SFTBackbone=Qwen3-8B, Setting=SFT2026.05 | 12.17 | 5.99 | 60.19 | 21.23 | 29.94 | 30.34 | — | |
| VerifyMAS-7B-SFTBackbone=Qwen2.5-7B, Setting=SFT2026.05 | 11.93 | 5.95 | 61.86 | 22.96 | 29.95 | 29.8 | — | |
| Qwen3-8B-Non-Thinking + SFTModel Scale=Medium-Scale, Training Protocol=SFT2025.09 | 9.68 | 5.73 | 64.79 | 38.96 | 20.37 | 20.36 | 21.41 | |
| Qwen3-8B-SFTBackbone=Qwen3-8B, Setting=SFT2026.05 | 9.68 | 5.73 | 64.79 | 38.96 | 20.37 | 20.36 | — | |
| DCLModel Scale=Small-Scale2025.09 | 8.33 | 5.3 | 22.93 | 20.23 | 24.73 | 27.7 | 12.61 | |
| DCLSetting=SFT2026.05 | 8.33 | 5.3 | 22.93 | 20.23 | 24.73 | 27.7 | — | |
| o3Model Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 7.86 | 2.27 | 40.31 | 23.27 | 22.37 | 16.76 | 20.24 | |
| Claude-Sonnet-4Model Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 7.68 | 2.34 | 40.73 | 15.51 | 21.21 | 16.55 | 18.16 | |
| GPT-4.1Model Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 7.44 | 2.27 | 37.48 | 11.12 | 20.65 | 15.75 | 15.27 | |
| Qwen2.5-7B-Instruct + GRPOModel Scale=Medium-Scale, Training Protocol=GRPO2025.09 | 7.11 | 2.77 | 35.43 | 14.86 | 17.21 | 10.54 | 14.87 | |
| Gemini-2.5-FlashModel Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 6.99 | 2.76 | 42.02 | 16.45 | 23.47 | 19.85 | 19.55 | |
| Gemini-2.5-ProModel Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 6.96 | 2.88 | 41.32 | 16.15 | 19.93 | 16.29 | 18.35 | |
| Qwen3-8B-Non-Thinking + GRPOModel Scale=Medium-Scale, Training Protocol=GRPO2025.09 | 6.94 | 2.82 | 45.91 | 17.39 | 20.89 | 15.15 | 17.15 | |
| Qwen2.5-14B-Instruct + Aegis-GRPOModel Scale=Medium-Scale, Training Protocol=GRPO2025.09 | 6.84 | 2.55 | 49.74 | 18.38 | 21.19 | 16.1 | 18.41 | |
| gpt-oss-120bModel Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 6.53 | 1.71 | 38.58 | 5.53 | 20.38 | 12.05 | 17.07 | |
| GPT-4o-miniModel Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 5.76 | 1.63 | 38.54 | 14.72 | 19.95 | 16.02 | 15.83 | |
| Qwen2.5-72B-InstructModel Scale=Large-Scale, Evaluation Protocol=Zero-shot2025.09 | 5.6 | 2.2 | 37.46 | 14.51 | 17.72 | 16.58 | 15.01 | |
| Qwen2.5-14B-InstructModel Scale=Medium-Scale, Evaluation Protocol=Zero-shot2025.09 | 5.47 | 2.2 | 35.78 | 12.71 | 20.24 | 5.91 | 13.99 | |
| w/o intentModel Scale=Small-Scale, Ablation Setting=w/o intent2025.09 | 5.43 | 6.83 | 13.7 | 13.9 | 22.67 | 23.8 | 10.16 | |
| only-mix headModel Scale=Small-Scale, Ablation Setting=only-mix head2025.09 | 5.17 | 4.2 | 24.33 | 22.6 | 25.2 | 26.8 | 12.42 | |
| Qwen2.5-7B-Instruct + SFTModel Scale=Medium-Scale, Training Protocol=SFT2025.09 | 5.05 | 2.8 | 60.03 | 22.7 | 19.61 | 16.9 | 17.99 | |
| Qwen2.5-7B-SFTBackbone=Qwen2.5-7B, Setting=SFT2026.05 | 5.05 | 2.8 | 60.03 | 22.7 | 19.61 | 16.9 | — | |
| Qwen2.5-7B-InstructModel Scale=Medium-Scale, Evaluation Protocol=Zero-shot2025.09 | 5.02 | 2.52 | 27.55 | 14.49 | 14.96 | 11.36 | 12.43 | |
| Qwen3-8B-ThinkingModel Scale=Medium-Scale2025.09 | 4.42 | 1.52 | 34.63 | 9.01 | 17.48 | 14.31 | 13.06 | |
| Qwen3-8B-Thinking + GRPOModel Scale=Medium-Scale, Training Protocol=GRPO2025.09 | 4.41 | 1.66 | 36.11 | 15.73 | 17.94 | 12.03 | 17.58 | |
| Qwen3-8B-Non-ThinkingModel Scale=Medium-Scale2025.09 | 3.96 | 1.4 | 21.34 | 8.16 | 15.81 | 13.89 | 10.12 | |
| w/o consistencyModel Scale=Small-Scale, Ablation Setting=w/o consistency2025.09 | 2.93 | 2.8 | 14.47 | 13.67 | 23.47 | 23.2 | 9.52 | |
| only-bilinearModel Scale=Small-Scale, Ablation Setting=only-bilinear2025.09 | 2.67 | 2.4 | 14.6 | 14 | 24.33 | 24.17 | 10.01 | |
| Random2025.09 | 0.33 | 0.21 | 4.54 | 3.56 | 11.23 | 11.15 | 4.08 |