Code Generation on MBPP (pass@1, ∆3)
73.33Pass@1MURPHY - MARS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MURPHY - MARSBackbone=Qwen3-4B, Rollouts=72, Iterations=32025.11 | 73.33 | 2.4 | |
| GRPOBackbone=Qwen3-4B, Rollouts=72, Iterations=32025.11 | 72.87 | 1.94 | |
| BaseBackbone=Qwen3-4B, Iterations=32025.11 | 70.93 | — | |
| MURPHY - MARSBackbone=Qwen3-1.7B, Rollouts=72, Iterations=32025.11 | 62 | 8.07 | |
| GRPOBackbone=Qwen3-1.7B, Rollouts=72, Iterations=32025.11 | 57.2 | 3.27 | |
| BaseBackbone=Qwen3-1.7B, Iterations=32025.11 | 53.93 | — | |
| MURPHY - MARSBackbone=Qwen3-4B, Rollouts=72, Iterations=12025.11 | 53.33 | — | |
| BaseBackbone=Qwen3-4B, Iterations=12025.11 | 52.13 | — | |
| GRPOBackbone=Qwen3-4B, Rollouts=72, Iterations=12025.11 | 51.73 | — | |
| GRPOBackbone=Qwen3-1.7B, Rollouts=72, Iterations=12025.11 | 45.93 | — | |
| MURPHY - MARSBackbone=Qwen3-1.7B, Rollouts=72, Iterations=12025.11 | 44.73 | — | |
| BaseBackbone=Qwen3-1.7B, Iterations=12025.11 | 43.47 | — | |
| MURPHY - MARSBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=32025.11 | 39.67 | 10.07 | |
| GRPOBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=32025.11 | 35.8 | 6.2 | |
| GRPOBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=12025.11 | 32.87 | — | |
| BaseBackbone=OLMo-2-1124-7B-Instruct, Iterations=32025.11 | 29.6 | — | |
| MURPHY - MARSBackbone=OLMo-2-1124-7B-Instruct, Rollouts=72, Iterations=12025.11 | 29.33 | — | |
| BaseBackbone=OLMo-2-1124-7B-Instruct, Iterations=12025.11 | 19.9 | — |