# Code Generation on BigCodeBench (pass@1, ∆3)

Source: [Code Generation on BigCodeBench (pass@1, ∆3)](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-pass-at-1-3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:11:15.604Z

Evaluation of model performance on the BigCodeBench dataset for Code Generation tasks, measured using pass@1 and ∆3 metrics.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Code Generation |
| Dataset (including any stated split) | BigCodeBench |
| Metric | pass@1 |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 18 of 18 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [pass@1](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-pass-at-1-3.md?metric=pass%401>): Higher is better; unit: PERCENT.

- [∆3](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-pass-at-1-3.md?metric=%E2%88%863>): Higher is better; unit: SCALAR.

## Recorded results

| Method | pass@1 | Result date | Report paper |
| --- | --- | --- | --- |
| MURPHY - MARS | 41.44 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 39.64 | Not recorded | 691414712828b7cc6937e961 |
| Base | 36.03 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 22.52 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 20.95 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 20.49 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 20.25 | Not recorded | 691414712828b7cc6937e961 |
| Base | 18.24 | Not recorded | 691414712828b7cc6937e961 |
| Base | 17.56 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 10.58 | Not recorded | 691414712828b7cc6937e961 |
| Base | 7.2 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 6.98 | Not recorded | 691414712828b7cc6937e961 |
| Base | 3.72 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 3.38 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 2.7 | Not recorded | 691414712828b7cc6937e961 |
| MURPHY - MARS | 1.8 | Not recorded | 691414712828b7cc6937e961 |
| Base | 1.35 | Not recorded | 691414712828b7cc6937e961 |
| GRPO | 0.68 | Not recorded | 691414712828b7cc6937e961 |

### MURPHY - MARS

pass@1: 41.44. Result ID: 699a6fdd5698665ec4f892ef.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### GRPO

pass@1: 39.64. Result ID: 699a6fdd5698665ec4f892ed.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### Base

pass@1: 36.03. Result ID: 699a6fdd5698665ec4f892eb.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

pass@1: 22.52. Result ID: 699a6fdd5698665ec4f892ee.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### GRPO

pass@1: 20.95. Result ID: 699a6fdd5698665ec4f892ec.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### GRPO

pass@1: 20.49. Result ID: 699a6fdd5698665ec4f892e7.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

pass@1: 20.25. Result ID: 699a6fdd5698665ec4f892e9.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### Base

pass@1: 18.24. Result ID: 699a6fdd5698665ec4f892e5.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Iterations":"3"}. This record does not assert independent reproduction.

### Base

pass@1: 17.56. Result ID: 699a6fdd5698665ec4f892ea.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Iterations":"1"}. This record does not assert independent reproduction.

### GRPO

pass@1: 10.58. Result ID: 699a6fdd5698665ec4f892e6.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

pass@1: 7.2. Result ID: 699a6fdd5698665ec4f892e4.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Iterations":"1"}. This record does not assert independent reproduction.

### MURPHY - MARS

pass@1: 6.98. Result ID: 699a6fdd5698665ec4f892e8.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-1.7B","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

pass@1: 3.72. Result ID: 699a6fdd5698665ec4f892f1.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

pass@1: 3.38. Result ID: 699a6fdd5698665ec4f892f5.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### GRPO

pass@1: 2.7. Result ID: 699a6fdd5698665ec4f892f3.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"3"}. This record does not assert independent reproduction.

### MURPHY - MARS

pass@1: 1.8. Result ID: 699a6fdd5698665ec4f892f4.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

### Base

pass@1: 1.35. Result ID: 699a6fdd5698665ec4f892f0.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Iterations":"1"}. This record does not assert independent reproduction.

### GRPO

pass@1: 0.68. Result ID: 699a6fdd5698665ec4f892f2.

[Reporting paper](<https://www.sota2.com/research/paper/691414712828b7cc6937e961.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"OLMo-2-1124-7B-Instruct","Rollouts":"72","Iterations":"1"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.