# Code Generation on MBPP subset of 500 (test) Source: [Code Generation on MBPP subset of 500 (test)]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:57:21.016Z Evaluation of model performance on the specific test subset of 500 problems from the Mostly Basic Python Programming (MBPP) dataset for code generation tasks, typically measured using Pass@1. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Code Generation | | Dataset (including any stated split) | MBPP subset of 500 (test) | | Metric | Pass@1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 | Result date | Report paper | | --- | --- | --- | --- | | Magicoder-S-DS | 64.6 | Not recorded | 68fa738f1e209131cea5d375 | | WaveCoder-Ultra-6.7B | 64.4 | Not recorded | 68fa738f1e209131cea5d375 | | WaveCoder-Pro-6.7B | 63.6 | Not recorded | 68fa738f1e209131cea5d375 | | WaveCoder-DS-6.7B | 62.8 | Not recorded | 68fa738f1e209131cea5d375 | | DeepseekCoder-instruct\* | 62.8 | Not recorded | 68fa738f1e209131cea5d375 | | DeepseekCoder | 60.6 | Not recorded | 68fa738f1e209131cea5d375 | | Magicoder-DS | 60.4 | Not recorded | 68fa738f1e209131cea5d375 | | ChatGPT | 52.2 | Not recorded | 68fa738f1e209131cea5d375 | | WizardCoder | 51.8 | Not recorded | 68fa738f1e209131cea5d375 | | WaveCoder-SC-15B | 51 | Not recorded | 68fa738f1e209131cea5d375 | | WaveCoder-CL-13B | 49.6 | Not recorded | 68fa738f1e209131cea5d375 | | CodeLLaMa-instruct | 49.4 | Not recorded | 68fa738f1e209131cea5d375 | | WaveCoder-CL-7B | 47.2 | Not recorded | 68fa738f1e209131cea5d375 | | CodeLLaMa | 47 | Not recorded | 68fa738f1e209131cea5d375 | | CodeLLaMa-instruct | 44.4 | Not recorded | 68fa738f1e209131cea5d375 | | OctoCoder | 43.5 | Not recorded | 68fa738f1e209131cea5d375 | | StarCoder | 43.3 | Not recorded | 68fa738f1e209131cea5d375 | | CodeLLaMa | 41.4 | Not recorded | 68fa738f1e209131cea5d375 | ### Magicoder-S-DS Pass@1: 64.6. Result ID: 699300283a6c31484912f5df. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"6.7B","Base Model":"DeepseekCoder","InsT Data":"185K","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### WaveCoder-Ultra-6.7B Pass@1: 64.4. Result ID: 699300283a6c31484912f5e0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"6.7B","Base Model":"DeepseekCoder","InsT Data":"130K","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### WaveCoder-Pro-6.7B Pass@1: 63.6. Result ID: 699300283a6c31484912f5dd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"6.7B","Base Model":"DeepseekCoder","InsT Data":"20K","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### WaveCoder-DS-6.7B Pass@1: 62.8. Result ID: 699300283a6c31484912f5dc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"6.7B","Base Model":"DeepseekCoder","InsT Data":"20K","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### DeepseekCoder-instruct\* Pass@1: 62.8. Result ID: 699300283a6c31484912f5de. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"6.7B","Base Model":"DeepseekCoder","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### DeepseekCoder Pass@1: 60.6. Result ID: 699300283a6c31484912f5da. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Params":"6.7B","InsT Data":"None","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### Magicoder-DS Pass@1: 60.4. Result ID: 699300283a6c31484912f5db. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"6.7B","Base Model":"DeepseekCoder","InsT Data":"75K","Decoding Strategy":"greedy"}. This record does not assert independent reproduction. ### ChatGPT Pass@1: 52.2. Result ID: 699300283a6c31484912f5cf. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### WizardCoder Pass@1: 51.8. Result ID: 699300283a6c31484912f5d2. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Params":"15B","Base Model":"StarCoder","InsT Data":"78K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### WaveCoder-SC-15B Pass@1: 51. Result ID: 699300283a6c31484912f5d3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"15B","Base Model":"StarCoder","InsT Data":"20K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### WaveCoder-CL-13B Pass@1: 49.6. Result ID: 699300283a6c31484912f5d9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"13B","Base Model":"CodeLLaMa","InsT Data":"20K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### CodeLLaMa-instruct Pass@1: 49.4. Result ID: 699300283a6c31484912f5d8. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Params":"13B","Base Model":"CodeLLaMa","InsT Data":"14K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### WaveCoder-CL-7B Pass@1: 47.2. Result ID: 699300283a6c31484912f5d6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"7B","Base Model":"CodeLLaMa","InsT Data":"20K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### CodeLLaMa Pass@1: 47. Result ID: 699300283a6c31484912f5d7. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Params":"13B","InsT Data":"None","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### CodeLLaMa-instruct Pass@1: 44.4. Result ID: 699300283a6c31484912f5d5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Params":"7B","Base Model":"CodeLLaMa","InsT Data":"14K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### OctoCoder Pass@1: 43.5. Result ID: 699300283a6c31484912f5d1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"15B","Base Model":"StarCoder","InsT Data":"13K","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### StarCoder Pass@1: 43.3. Result ID: 699300283a6c31484912f5d0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Params":"15B","InsT Data":"None","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. ### CodeLLaMa Pass@1: 41.4. Result ID: 699300283a6c31484912f5d4. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Params":"7B","InsT Data":"None","Decoding Strategy":"n=200"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.