# Functional Code Generation on HumanEval 8 instruct cot (test) Source: [Functional Code Generation on HumanEval 8 instruct cot (test)]() Collection index: [llms.txt]() Data updated: 2026-05-11T10:11:09.521Z Evaluation of model performance on functional code generation using the HumanEval 8 instruct cot test set, typically measured by Pass@k metrics. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Functional Code Generation | | Dataset (including any stated split) | HumanEval 8 instruct cot (test) | | Metric | Pass@1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1](): Higher is better; unit: PERCENT. - [Pass@8](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 | Result date | Report paper | | --- | --- | --- | --- | | gpt-oss-20b | 94.7 | Not recorded | 6a0162badbc836b87d82332a | | Qwen3-4B | 93.3 | Not recorded | 6a0162badbc836b87d82332a | | GPT-5 nano | 84.8 | Not recorded | 6a0162badbc836b87d82332a | | Qwen3-8B | 80.1 | Not recorded | 6a0162badbc836b87d82332a | | gemma-3-12b-it | 76.2 | Not recorded | 6a0162badbc836b87d82332a | | gemma-3-4b-it | 64.1 | Not recorded | 6a0162badbc836b87d82332a | | Moonlight-16B-A3B-Instruct | 61.1 | Not recorded | 6a0162badbc836b87d82332a | | EngGPT2-16B-A3B | 45.2 | Not recorded | 6a0162badbc836b87d82332a | | Llama-3.2-3B-Instruct | 33.9 | Not recorded | 6a0162badbc836b87d82332a | | Llama-3.1-8B-Instruct | 28.7 | Not recorded | 6a0162badbc836b87d82332a | | LLaMAntino-3-ANITA-8B-Inst-DPO-ITA | 26 | Not recorded | 6a0162badbc836b87d82332a | | Ministral-3-8B-Instruct-2512-BF16 | 25.1 | Not recorded | 6a0162badbc836b87d82332a | | FastwebMIIA-7B | 18.5 | Not recorded | 6a0162badbc836b87d82332a | | deepseek-moe-16b-chat | 17.8 | Not recorded | 6a0162badbc836b87d82332a | | Velvet-14B | 12.7 | Not recorded | 6a0162badbc836b87d82332a | | Minerva-7B-instruct-v1.0 | 5 | Not recorded | 6a0162badbc836b87d82332a | ### gpt-oss-20b Pass@1: 94.7. Result ID: 6a01ab3c47f34ce924e41e73. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Qwen3-4B Pass@1: 93.3. Result ID: 6a01ab3c47f34ce924e41e6a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### GPT-5 nano Pass@1: 84.8. Result ID: 6a01ab3c47f34ce924e41e71. [Reporting paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Qwen3-8B Pass@1: 80.1. Result ID: 6a01ab3c47f34ce924e41e6b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### gemma-3-12b-it Pass@1: 76.2. Result ID: 6a01ab3c47f34ce924e41e6f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### gemma-3-4b-it Pass@1: 64.1. Result ID: 6a01ab3c47f34ce924e41e6e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Moonlight-16B-A3B-Instruct Pass@1: 61.1. Result ID: 6a01ab3c47f34ce924e41e74. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### EngGPT2-16B-A3B Pass@1: 45.2. Result ID: 6a01ab3c47f34ce924e41e65. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Llama-3.2-3B-Instruct Pass@1: 33.9. Result ID: 6a01ab3c47f34ce924e41e6c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Llama-3.1-8B-Instruct Pass@1: 28.7. Result ID: 6a01ab3c47f34ce924e41e6d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### LLaMAntino-3-ANITA-8B-Inst-DPO-ITA Pass@1: 26. Result ID: 6a01ab3c47f34ce924e41e69. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Ministral-3-8B-Instruct-2512-BF16 Pass@1: 25.1. Result ID: 6a01ab3c47f34ce924e41e70. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### FastwebMIIA-7B Pass@1: 18.5. Result ID: 6a01ab3c47f34ce924e41e66. [Reporting paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### deepseek-moe-16b-chat Pass@1: 17.8. Result ID: 6a01ab3c47f34ce924e41e72. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Velvet-14B Pass@1: 12.7. Result ID: 6a01ab3c47f34ce924e41e68. [Reporting paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. ### Minerva-7B-instruct-v1.0 Pass@1: 5. Result ID: 6a01ab3c47f34ce924e41e67. [Reporting paper]() Recorded evaluation configuration: {"Prompting":"Chain-of-thought","Instruction-following":"true"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.