# Code Generation on HumanEval (var@10, var@100, Pres.) Source: [Code Generation on HumanEval (var@10, var@100, Pres.)]() Collection index: [llms.txt]() Data updated: 2026-06-05T09:51:39.499Z Evaluation of model performance on the HumanEval dataset for code generation, reporting functional correctness metrics including variation rates at k=10 and k=100, and a metric labeled 'Pres.'. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Code Generation | | Dataset (including any stated split) | HumanEval | | Metric | Variation Rate @ k=10 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Variation Rate @ k=10](): Higher is better; unit: PERCENT. - [Variation Rate @ k=100](): Higher is better; unit: PERCENT. - [Preservation Rate](): Higher is better; unit: PERCENT. ## Recorded results | Method | Variation Rate @ k=10 | Result date | Report paper | | --- | --- | --- | --- | | Dense | 35.88 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 35.72 | Not recorded | 6a2258ae86173ad9c1468c67 | | Dense | 34.71 | Not recorded | 6a2258ae86173ad9c1468c67 | | Dense | 33.89 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 31.97 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 29.44 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 29.33 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 27.71 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 27.12 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 26.98 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 26.96 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 26.62 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 26.48 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 23.64 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 23 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 22.18 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 20.53 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 20.24 | Not recorded | 6a2258ae86173ad9c1468c67 | | SecRL-Prune | 18.91 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 18.8 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 18.51 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 16.72 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 16.46 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 14.27 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 12.43 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 9.83 | Not recorded | 6a2258ae86173ad9c1468c67 | | PruneNet | 8.57 | Not recorded | 6a2258ae86173ad9c1468c67 | ### Dense Variation Rate @ k=10: 35.88. Result ID: 6a229b59fbc373592c212ded. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"0%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 35.72. Result ID: 6a229b59fbc373592c212ddc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"10%"}. This record does not assert independent reproduction. ### Dense Variation Rate @ k=10: 34.71. Result ID: 6a229b59fbc373592c212ddb. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"0%"}. This record does not assert independent reproduction. ### Dense Variation Rate @ k=10: 33.89. Result ID: 6a229b59fbc373592c212de4. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"0%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 31.97. Result ID: 6a229b59fbc373592c212de5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"10%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 29.44. Result ID: 6a229b59fbc373592c212dee. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"10%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 29.33. Result ID: 6a229b59fbc373592c212ddd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"10%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 27.71. Result ID: 6a229b59fbc373592c212de0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"25%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 27.12. Result ID: 6a229b59fbc373592c212dde. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"20%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 26.98. Result ID: 6a229b59fbc373592c212de7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"20%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 26.96. Result ID: 6a229b59fbc373592c212df0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"20%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 26.62. Result ID: 6a229b59fbc373592c212de6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"10%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 26.48. Result ID: 6a229b59fbc373592c212de2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"30%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 23.64. Result ID: 6a229b59fbc373592c212de9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"25%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 23. Result ID: 6a229b59fbc373592c212df2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"25%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 22.18. Result ID: 6a229b59fbc373592c212def. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"10%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 20.53. Result ID: 6a229b59fbc373592c212df4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"30%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 20.24. Result ID: 6a229b59fbc373592c212ddf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"20%"}. This record does not assert independent reproduction. ### SecRL-Prune Variation Rate @ k=10: 18.91. Result ID: 6a229b59fbc373592c212deb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"30%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 18.8. Result ID: 6a229b59fbc373592c212de8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"20%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 18.51. Result ID: 6a229b59fbc373592c212de1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"25%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 16.72. Result ID: 6a229b59fbc373592c212de3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-Coder-7B-Instruct","Compression Ratio":"30%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 16.46. Result ID: 6a229b59fbc373592c212df1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"20%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 14.27. Result ID: 6a229b59fbc373592c212dea. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"25%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 12.43. Result ID: 6a229b59fbc373592c212df3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"25%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 9.83. Result ID: 6a229b59fbc373592c212dec. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Instruct","Compression Ratio":"30%"}. This record does not assert independent reproduction. ### PruneNet Variation Rate @ k=10: 8.57. Result ID: 6a229b59fbc373592c212df5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"CodeLlama-7B-Python","Compression Ratio":"30%"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.