Code Generation on BigCodeBench (pass@1, ∆3)
41.44pass@1MURPHY - MARS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MURPHY - MARSBackbone=Qwen3-4B, Rollouts=72, Iterations=32025.11 | 41.44 | 5.41 | |
| GRPOBackbone=Qwen3-4B, Rollouts=72, Iterations=32025.11 | 39.64 | 3.61 | |
| BaseBackbone=Qwen3-4B, Iterations=32025.11 | 36.03 | — | |
| MURPHY - MARSBackbone=Qwen3-4B, Rollouts=72, Iterations=12025.11 | 22.52 | — | |
| GRPOBackbone=Qwen3-4B, Rollouts=72, Iterations=12025.11 | 20.95 | — | |
| GRPOBackbone=Qwen3-1.7B, Rollouts=72, Iterations=32025.11 | 20.49 | 2.25 | |
| MURPHY - MARSBackbone=Qwen3-1.7B, Rollouts=72, Iterations=32025.11 | 20.25 | 2.01 | |
| BaseBackbone=Qwen3-1.7B, Iterations=32025.11 | 18.24 | — | |
| BaseBackbone=Qwen3-4B, Iterations=12025.11 | 17.56 | — | |
| GRPOBackbone=Qwen3-1.7B, Rollouts=72, Iterations=12025.11 | 10.58 | — | |
| BaseBackbone=Qwen3-1.7B, Iterations=12025.11 | 7.2 | — | |
| MURPHY - MARSBackbone=Qwen3-1.7B, Rollouts=72, Iterations=12025.11 | 6.98 | — | |
| BaseBackbone=OLMo-2-1124-7B-Instruct, Iterations=32025.11 | 3.72 | — | |
| MURPHY - MARSBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=32025.11 | 3.38 | -0.34 | |
| GRPOBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=32025.11 | 2.7 | -1.02 | |
| MURPHY - MARSBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=12025.11 | 1.8 | — | |
| BaseBackbone=OLMo-2-1124-7B-Instruct, Iterations=12025.11 | 1.35 | — | |
| GRPOBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=12025.11 | 0.68 | — |