Debugging on OR-Debug-Bench (450 stratified samples)
100RRQwen3-8B-GRPO
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen3-8B-GRPOType=Local, Evaluation Protocol=multi-attempt2026.01 | 100 | 95.3 | 62.4 | 2.25 | |
| Qwen3-8B-CurriculumType=Local, Evaluation Protocol=multi-attempt2026.01 | 100 | 94 | 61.7 | 2.22 | |
| Qwen3-8B-DAPOType=Local, Evaluation Protocol=multi-attempt2026.01 | 100 | 93.8 | 60.4 | 2.31 | |
| claude-sonnet-4Type=API, Evaluation Protocol=multi-attempt2026.01 | 100 | 86.2 | 50.1 | 3.71 | |
| gpt-5-miniType=API, Evaluation Protocol=multi-attempt2026.01 | 100 | 66.9 | 37.6 | 4.74 | |
| GRPOBackbone=Qwen3-8B, Training Pipeline=SFT→GRPO2026.01 | 100 | 95.3 | 62.4 | 2.25 | |
| CurriculumBackbone=Qwen3-8B, Training Pipeline=SFT→GRPO2026.01 | 100 | 94 | 61.7 | 2.22 | |
| DAPOBackbone=Qwen3-8B, Training Pipeline=SFT→GRPO2026.01 | 100 | 93.8 | 60.4 | 2.31 | |
| GRPOBackbone=Llama-3.1-8B, Training Pipeline=SFT→GRPO2026.01 | 100 | 97.3 | 60.3 | 1.82 | |
| SFTBackbone=Llama-3.1-8B, Training Pipeline=SFT2026.01 | 100 | 97.1 | 60.4 | 1.85 | |
| claude-sonnet-4type=API model baseline, training=no training2026.01 | 100 | 86.2 | 50.1 | 3.71 | |
| Qwen3-8B-SFTType=Local, Evaluation Protocol=multi-attempt2026.01 | 99.8 | 93.1 | 60.8 | 2.34 | |
| o1Type=API, Evaluation Protocol=multi-attempt2026.01 | 99.8 | 82.9 | 47.8 | 3.78 | |
| gpt-5.2-chatType=API, Evaluation Protocol=multi-attempt2026.01 | 99.8 | 81.8 | 40.9 | 3.72 | |
| SFTBackbone=Qwen3-8B, Training Pipeline=SFT2026.01 | 99.8 | 93.1 | 60.8 | 2.34 | |
| o1type=API model baseline, training=no training2026.01 | 99.8 | 82.9 | 47.8 | 3.78 | |
| gpt-5.2-chattype=API model baseline, training=no training2026.01 | 99.8 | 81.8 | 40.9 | 3.72 | |
| GRPOBackbone=DeepSeek-R1-Distill-Qwen3-8B, Training Pipeline=SFT→GRPO2026.01 | 99.6 | 88.9 | 52.2 | 3.3 | |
| claude-haiku-4.5Type=API, Evaluation Protocol=multi-attempt2026.01 | 99.3 | 86 | 53.1 | 3.89 | |
| DeepSeek-V3.2Type=API, Evaluation Protocol=multi-attempt2026.01 | 99.3 | 58.9 | 44.8 | 4.86 | |
| SFTBackbone=DeepSeek-R1-Distill-Qwen3-8B, Training Pipeline=SFT2026.01 | 99.3 | 88.7 | 52.2 | 3.3 | |
| DeepSeek-V3.2type=API model baseline, training=no training2026.01 | 99.3 | 58.9 | 44.8 | 4.86 | |
| DeepSeek-R1Type=API, Evaluation Protocol=multi-attempt2026.01 | 99.1 | 56.7 | 34.5 | 5.08 | |
| qwen2.5-maxType=API, Evaluation Protocol=multi-attempt2026.01 | 99.1 | 54.9 | 42.6 | 5.97 | |
| DeepSeek-R1type=API model baseline, training=no training2026.01 | 99.1 | 56.7 | 34.5 | 5.08 | |
| qwen2.5-32bType=API, Evaluation Protocol=multi-attempt2026.01 | 98.9 | 61.1 | 32 | 4.98 | |
| claude-3.7-sonnetType=API, Evaluation Protocol=multi-attempt2026.01 | 98.9 | 31.6 | 46.8 | 8.06 | |
| o4-miniType=API, Evaluation Protocol=multi-attempt2026.01 | 97.8 | 86.2 | 47.8 | 3.15 | |
| qwen2.5-7bType=API, Evaluation Protocol=multi-attempt2026.01 | 97.8 | 77.8 | 40.1 | 4.36 | |
| o4-minitype=API model baseline, training=no training2026.01 | 97.8 | 86.2 | 47.8 | 3.15 | |
| o3Type=API, Evaluation Protocol=multi-attempt2026.01 | 96.7 | 75.8 | 50.9 | 4.23 | |
| gpt-4.1Type=API, Evaluation Protocol=multi-attempt2026.01 | 94.4 | 71.6 | 36.2 | 4.41 | |
| claude-opus-4Type=API, Evaluation Protocol=multi-attempt2026.01 | 94.2 | 76.9 | 49 | 3.92 | |
| Llama-3.3-70BType=API, Evaluation Protocol=multi-attempt2026.01 | 93.8 | 60.9 | 46.9 | 4.81 | |
| Llama-3.3-70Btype=API model baseline, training=no training2026.01 | 93.8 | 60.9 | 46.9 | 4.81 | |
| gpt-4.1-miniType=API, Evaluation Protocol=multi-attempt2026.01 | 93.1 | 49.8 | 26 | 6.04 | |
| qwen2.5-14bType=API, Evaluation Protocol=multi-attempt2026.01 | 88.9 | 53.6 | 35.4 | 6.32 | |
| gemini-2.0-flashType=API, Evaluation Protocol=multi-attempt2026.01 | 85.6 | 52.4 | 18.3 | 5.63 | |
| gemini-2.5-flashType=API, Evaluation Protocol=multi-attempt2026.01 | 84.2 | 70.7 | 19.2 | 3.23 | |
| gemini-2.5-flashtype=API model baseline, training=no training2026.01 | 84.2 | 70.7 | 19.2 | 3.23 | |
| gemini-2.5-proType=API, Evaluation Protocol=multi-attempt2026.01 | 62.9 | 62.7 | 52.5 | 0.83 | |
| kimi-k2Type=API, Evaluation Protocol=multi-attempt2026.01 | 55.3 | 40.4 | 25.6 | 2.48 |