# Code Generation on MBPP (pass@1, ∆3)

Source: [Code Generation on MBPP (pass@1, ∆3)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1-3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:57:29.467Z

Evaluation of model performance on the Mostly Basic Python Programming (MBPP) dataset for code generation tasks, specifically reporting Pass@1 and the Delta 3 metric.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Code Generation |
| Dataset (including any stated split) | MBPP |
| Metric | Pass@1 |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 18 of 18 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Pass@1](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1-3.md?metric=Pass%401>): Higher is better; unit: PERCENT.

- [Delta 3](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1-3.md?metric=Delta%203>): Higher is better; unit: SCALAR.

## Recorded results

| Method | Pass@1 | Result date | Report paper |
| --- | --- | --- | --- |
| MURPHY - MARS | 73.33 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 72.87 | Not recorded | 691414712828b7cc6937e961 |
| Base | 70.93 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 62 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 57.2 | Not recorded | 691414712828b7cc6937e961 |
| Base | 53.93 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 53.33 | Not recorded | 691414712828b7cc6937e961 |
| Base | 52.13 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 51.73 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 45.93 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 44.73 | Not recorded | 691414712828b7cc6937e961 |
| Base | 43.47 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 39.67 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 35.8 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 32.87 | Not recorded | 691414712828b7cc6937e961 |
| Base | 29.6 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 29.33 | Not recorded | 691414712828b7cc6937e961 |
| Base | 19.9 | Not recorded | 691414712828b7cc6937e961 |

### MURPHY - MARS

Pass@1: 73.33. Result ID: 699a6fdd5698665ec4f892dd.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### GRPO

Pass@1: 72.87. Result ID: 699a6fdd5698665ec4f892db.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### Base

Pass@1: 70.93. Result ID: 699a6fdd5698665ec4f892d9.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

Pass@1: 62. Result ID: 699a6fdd5698665ec4f892d7.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### GRPO

Pass@1: 57.2. Result ID: 699a6fdd5698665ec4f892d5.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### Base

Pass@1: 53.93. Result ID: 699a6fdd5698665ec4f892d3.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

Pass@1: 53.33. Result ID: 699a6fdd5698665ec4f892dc.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

Pass@1: 52.13. Result ID: 699a6fdd5698665ec4f892d8.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Iterations":"1"}. This record does not assert independent reproduction.

### GRPO

Pass@1: 51.73. Result ID: 699a6fdd5698665ec4f892da.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### GRPO

Pass@1: 45.93. Result ID: 699a6fdd5698665ec4f892d4.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### MURPHY - MARS

Pass@1: 44.73. Result ID: 699a6fdd5698665ec4f892d6.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

Pass@1: 43.47. Result ID: 699a6fdd5698665ec4f892d2.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Iterations":"1"}. This record does not assert independent reproduction.

### MURPHY - MARS

Pass@1: 39.67. Result ID: 699a6fdd5698665ec4f892e3.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### GRPO

Pass@1: 35.8. Result ID: 699a6fdd5698665ec4f892e1.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### GRPO

Pass@1: 32.87. Result ID: 699a6fdd5698665ec4f892e0.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

Pass@1: 29.6. Result ID: 699a6fdd5698665ec4f892df.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

Pass@1: 29.33. Result ID: 699a6fdd5698665ec4f892e2.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

Pass@1: 19.9. Result ID: 699a6fdd5698665ec4f892de.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Iterations":"1"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.