Coding on BigCodeBench Hard
33.8Pass@1P-GRPO (Code+RM)
Evaluation results
| Method | Links | |
|---|---|---|
| P-GRPO (Code+RM)Size=7B-Inst, Training Setting=Training-based2026.05 | 33.8 | |
| GRPO (Code)Size=7B-Inst, Training Setting=Training-based2026.05 | 29.7 | |
| HASP-Evolve + RSSize=7B-Inst, Training Setting=Training-based, Backbone=Qwen2.5-7B-Instruct2026.05 | 28 | |
| HASP-Intervention (w. Teacher)Size=7B-Inst, Training Setting=Training-free / inference-time, Backbone=Qwen2.5-7B-Instruct2026.05 | 26 | |
| KodCode-RL Step 128Size=7B-Inst, Training Setting=Training-based2026.05 | 19.6 | |
| KodCode-RL Step 256Size=7B-Inst, Training Setting=Training-based2026.05 | 19.6 | |
| HASP-Intervention (PF-only)Size=7B-Inst, Training Setting=Training-free / inference-time, Backbone=Qwen2.5-7B-Instruct2026.05 | 19 | |
| Qwen2.5-7B-InstructSize=7B-Inst, Training Setting=Training-free / inference-time2026.05 | 16.9 | |
| AceCoderRMSize=7B-Inst, Training Setting=Training-based2026.05 | 16.9 | |
| AceCoderRuleSize=7B-Inst, Training Setting=Training-based2026.05 | 15.5 | |
| RA-Agent (multi-loop)Size=7B-Inst, Training Setting=Training-free / inference-time2026.05 | 14 | |
| Prompt-Only SkillsSize=7B-Inst, Training Setting=Training-free / inference-time2026.05 | 14 | |
| SFT (vanilla)Size=7B-Inst, Training Setting=Training-based2026.05 | 13 | |
| GPT-4o-miniSize=~8B, Training Setting=Training-free / inference-time2026.05 | 12 | |
| GPT-4oSize=~200B, Training Setting=Training-free / inference-time2026.05 | 9 |