# Verilog Generation on RTLLM v2 Source: [Verilog Generation on RTLLM v2]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:57:35.626Z Evaluation of Verilog code generation performance on the RTLLM v2 dataset, measured using Pass@N metrics (N=15, 50, 100, 200, 300). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Verilog Generation | | Dataset (including any stated split) | RTLLM v2 | | Metric | Pass@200 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 12 of 12 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@200](): Higher is better; unit: PERCENT. - [Pass@15](): Higher is better; unit: PERCENT. - [Pass@50](): Higher is better; unit: PERCENT. - [Pass@100](): Higher is better; unit: PERCENT. - [Pass@300](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@200 | Result date | Report paper | | --- | --- | --- | --- | | GPT-OSS-120B-IGR | 90 | Not recorded | 6978470aa2b28e761d87c95a | | DeepSeek-R1-FP4-MCTS | 90 | Not recorded | 6978470aa2b28e761d87c95a | | REvolution (Deepseek-V3) | 88 | Not recorded | 6978470aa2b28e761d87c95a | | GPT-OSS-120B-MCTS | 88 | Not recorded | 6978470aa2b28e761d87c95a | | DeepSeek-R1-FP4 | 86 | Not recorded | 6978470aa2b28e761d87c95a | | Siliconmind-7B | 84 | Not recorded | 6978470aa2b28e761d87c95a | | REvolution (Llama-3.3-70B) | 84 | Not recorded | 6978470aa2b28e761d87c95a | | Siliconmind-7B-MCTS | 84 | Not recorded | 6978470aa2b28e761d87c95a | | Gemini-2.0-Flash-MCTS | 82 | Not recorded | 6978470aa2b28e761d87c95a | | Gemini-2.0-Flash-IGR | 80 | Not recorded | 6978470aa2b28e761d87c95a | | Gemini-2.0-Flash | 78 | Not recorded | 6978470aa2b28e761d87c95a | | GPT-OSS-120B | 76 | Not recorded | 6978470aa2b28e761d87c95a | ### GPT-OSS-120B-IGR Pass@200: 90. Result ID: 6996223a5698665ec462ba9d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"STG":"false","Search Strategy":"Idea-Guided Refinement"}. This record does not assert independent reproduction. ### DeepSeek-R1-FP4-MCTS Pass@200: 90. Result ID: 6996223a5698665ec462ba9f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"STG":"false","Search Strategy":"Monte Carlo Tree Search"}. This record does not assert independent reproduction. ### REvolution (Deepseek-V3) Pass@200: 88. Result ID: 6996223a5698665ec462ba98. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"API Access":"true"}. This record does not assert independent reproduction. ### GPT-OSS-120B-MCTS Pass@200: 88. Result ID: 6996223a5698665ec462ba9e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"STG":"false","Search Strategy":"Monte Carlo Tree Search"}. This record does not assert independent reproduction. ### DeepSeek-R1-FP4 Pass@200: 86. Result ID: 6996223a5698665ec462ba97. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Siliconmind-7B Pass@200: 84. Result ID: 6996223a5698665ec462ba95. [Reporting paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### REvolution (Llama-3.3-70B) Pass@200: 84. Result ID: 6996223a5698665ec462ba99. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"API Access":"true"}. This record does not assert independent reproduction. ### Siliconmind-7B-MCTS Pass@200: 84. Result ID: 6996223a5698665ec462ba9c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"STG":"false","Search Strategy":"Monte Carlo Tree Search"}. This record does not assert independent reproduction. ### Gemini-2.0-Flash-MCTS Pass@200: 82. Result ID: 6996223a5698665ec462ba9b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"STG":"false","Search Strategy":"Monte Carlo Tree Search","API Access":"true"}. This record does not assert independent reproduction. ### Gemini-2.0-Flash-IGR Pass@200: 80. Result ID: 6996223a5698665ec462ba9a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"STG":"false","Search Strategy":"Idea-Guided Refinement","API Access":"true"}. This record does not assert independent reproduction. ### Gemini-2.0-Flash Pass@200: 78. Result ID: 6996223a5698665ec462ba94. [Reporting paper]() Recorded evaluation configuration: {"API Access":"true"}. This record does not assert independent reproduction. ### GPT-OSS-120B Pass@200: 76. Result ID: 6996223a5698665ec462ba96. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.