# Verilog Generation on VerilogEval v2 Source: [Verilog Generation on VerilogEval v2]() Collection index: [llms.txt]() Data updated: 2026-06-12T07:30:48.304Z Evaluation of model performance on generating Verilog code using the VerilogEval v2 dataset, measured using Pass@k metrics (k=15, 50, 100, 200, 300). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Verilog Generation | | Dataset (including any stated split) | VerilogEval v2 | | Metric | Pass@1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 41 of 41 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1](): Higher is better; unit: PERCENT. - [Pass@15](): Higher is better; unit: PERCENT. - [Pass@50](): Higher is better; unit: PERCENT. - [Pass@100](): Higher is better; unit: PERCENT. - [Pass@200](): Higher is better; unit: PERCENT. - [Pass@300](): Higher is better; unit: PERCENT. - [Pass@5](): Higher is better; unit: PERCENT. - [Pass@10](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 | Result date | Report paper | | --- | --- | --- | --- | | GPT-OSS-120B | 89.6 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | Claude Opus 4.7 | 86.9 | Not recorded | 6a0556e7dbc836b87d828ae4 | | GPT-5.5 | 84.7 | Not recorded | 6a0556e7dbc836b87d828ae4 | | SiliconMind-V1 | 82 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | SiliconMind-V1 | 81 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | STG | 80.2 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | ChipMATE-Agents-9B | 80.1 | Not recorded | 6a0556e7dbc836b87d828ae4 | | STG | 80 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | DeepSeek R1 | 77.5 | Not recorded | 6a0556e7dbc836b87d828ae4 | | ChipMATE-Verilog-9B | 75.3 | Not recorded | 6a0556e7dbc836b87d828ae4 | | ChipMATE-Agents-4B | 75 | Not recorded | 6a0556e7dbc836b87d828ae4 | | SiliconMind-V1 | 73.9 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | CodeV-R1 | 73.2 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | STG | 70.5 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | CodeV-R1 | 68.8 | Not recorded | 6a0556e7dbc836b87d828ae4 | | DeepSeek Coder | 68.5 | Not recorded | 6a0556e7dbc836b87d828ae4 | | ChipMATE-Verilog-4B | 67.4 | Not recorded | 6a0556e7dbc836b87d828ae4 | | DeepSeek V4 | 67.3 | Not recorded | 6a0556e7dbc836b87d828ae4 | | VeriRL (paper) | 67.2 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | CodeV-R1 (distill) | 65.2 | Not recorded | 6a0556e7dbc836b87d828ae4 | | GPT-4o | 64.1 | Not recorded | 6a0556e7dbc836b87d828ae4 | | VeriRL (reproduced) | 60.8 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | Qwen3-8B | 52.5 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | Qwen3.5-9B | 48.5 | Not recorded | 6a0556e7dbc836b87d828ae4 | | Qwen3.5-4B | 41.7 | Not recorded | 6a0556e7dbc836b87d828ae4 | | Qwen2.5-C-7B-Instruct | 33.6 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | | Qwen3-4B-Thinking | 21.4 | Not recorded | 6a2b92bcd33dfb23e783b1a2 | ### GPT-OSS-120B Pass@1: 89.6. Result ID: 6a2bb59cfbc373592c29c2c1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Teacher","SFT":"null","RL":"null"}. This record does not assert independent reproduction. ### Claude Opus 4.7 Pass@1: 86.9. Result ID: 6a057af647f34ce924ec2341. [Reporting paper]() Recorded evaluation configuration: {"Type":"Foundation Models","Size":null}. This record does not assert independent reproduction. ### GPT-5.5 Pass@1: 84.7. Result ID: 6a057af647f34ce924ec2340. [Reporting paper]() Recorded evaluation configuration: {"Type":"Foundation Models","Size":null}. This record does not assert independent reproduction. ### SiliconMind-V1 Pass@1: 82. Result ID: 6a2bb59cfbc373592c29c2ca. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen3-4B-Thinking","SFT":"true","RL":"false"}. This record does not assert independent reproduction. ### SiliconMind-V1 Pass@1: 81. Result ID: 6a2bb59cfbc373592c29c2cc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen3-8B","SFT":"true","RL":"false"}. This record does not assert independent reproduction. ### STG Pass@1: 80.2. Result ID: 6a2bb59cfbc373592c29c2cd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen3-8B","SFT":"true","RL":"false"}. This record does not assert independent reproduction. ### ChipMATE-Agents-9B Pass@1: 80.1. Result ID: 6a057af647f34ce924ec234c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"ChipMATE (Ours)","Size":"9B"}. This record does not assert independent reproduction. ### STG Pass@1: 80. Result ID: 6a2bb59cfbc373592c29c2cb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen3-4B-Thinking","SFT":"true","RL":"false"}. This record does not assert independent reproduction. ### DeepSeek R1 Pass@1: 77.5. Result ID: 6a057af647f34ce924ec2344. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"Foundation Models","Size":"671B"}. This record does not assert independent reproduction. ### ChipMATE-Verilog-9B Pass@1: 75.3. Result ID: 6a057af647f34ce924ec234b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"ChipMATE (Ours)","Size":"9B"}. This record does not assert independent reproduction. ### ChipMATE-Agents-4B Pass@1: 75. Result ID: 6a057af647f34ce924ec234a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"ChipMATE (Ours)","Size":"4B"}. This record does not assert independent reproduction. ### SiliconMind-V1 Pass@1: 73.9. Result ID: 6a2bb59cfbc373592c29c2c8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen2.5-C-7B-Instruct","SFT":"true","RL":"false"}. This record does not assert independent reproduction. ### CodeV-R1 Pass@1: 73.2. Result ID: 6a2bb59cfbc373592c29c2c5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen2.5-C-7B-Instruct","SFT":"true","RL":"true"}. This record does not assert independent reproduction. ### STG Pass@1: 70.5. Result ID: 6a2bb59cfbc373592c29c2c9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen2.5-C-7B-Instruct","SFT":"true","RL":"false"}. This record does not assert independent reproduction. ### CodeV-R1 Pass@1: 68.8. Result ID: 6a057af647f34ce924ec2346. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"Specialized Models","Size":"7B"}. This record does not assert independent reproduction. ### DeepSeek Coder Pass@1: 68.5. Result ID: 6a057af647f34ce924ec2342. [Reporting paper]() Recorded evaluation configuration: {"Type":"Foundation Models","Size":"236B"}. This record does not assert independent reproduction. ### ChipMATE-Verilog-4B Pass@1: 67.4. Result ID: 6a057af647f34ce924ec2349. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"ChipMATE (Ours)","Size":"4B"}. This record does not assert independent reproduction. ### DeepSeek V4 Pass@1: 67.3. Result ID: 6a057af647f34ce924ec2343. [Reporting paper]() Recorded evaluation configuration: {"Type":"Foundation Models","Size":"1.6T"}. This record does not assert independent reproduction. ### VeriRL (paper) Pass@1: 67.2. Result ID: 6a2bb59cfbc373592c29c2c6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen2.5-C-7B-Instruct","SFT":"true","RL":"true"}. This record does not assert independent reproduction. ### CodeV-R1 (distill) Pass@1: 65.2. Result ID: 6a057af647f34ce924ec2345. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Type":"Specialized Models","Size":"7B"}. This record does not assert independent reproduction. ### GPT-4o Pass@1: 64.1. Result ID: 6a057af647f34ce924ec233f. [Reporting paper]() Recorded evaluation configuration: {"Type":"Foundation Models","Size":null}. This record does not assert independent reproduction. ### VeriRL (reproduced) Pass@1: 60.8. Result ID: 6a2bb59cfbc373592c29c2c7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Fine-tuned","Base Model":"Qwen2.5-C-7B-Instruct","SFT":"true","RL":"true"}. This record does not assert independent reproduction. ### Qwen3-8B Pass@1: 52.5. Result ID: 6a2bb59cfbc373592c29c2c4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Base","SFT":"null","RL":"null"}. This record does not assert independent reproduction. ### Qwen3.5-9B Pass@1: 48.5. Result ID: 6a057af647f34ce924ec2348. [Reporting paper]() Recorded evaluation configuration: {"Type":"Base Models","Size":"9B"}. This record does not assert independent reproduction. ### Qwen3.5-4B Pass@1: 41.7. Result ID: 6a057af647f34ce924ec2347. [Reporting paper]() Recorded evaluation configuration: {"Type":"Base Models","Size":"4B"}. This record does not assert independent reproduction. ### Qwen2.5-C-7B-Instruct Pass@1: 33.6. Result ID: 6a2bb59cfbc373592c29c2c2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Base","SFT":"null","RL":"null"}. This record does not assert independent reproduction. ### Qwen3-4B-Thinking Pass@1: 21.4. Result ID: 6a2bb59cfbc373592c29c2c3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Role":"Base","SFT":"null","RL":"null"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.