Verilog Generation on VerilogEval v2
89.6Pass@1GPT-OSS-120B
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GPT-OSS-120BRole=Teacher, SFT=null, RL=null2026.06 | 89.6 | — | — | — | — | — | 96.7 | 97.6 | |
| Claude Opus 4.7Type=Foundation Models2026.05 | 86.9 | — | — | — | — | — | 90.4 | — | |
| GPT-5.5Type=Foundation Models2026.05 | 84.7 | — | — | — | — | — | 90.4 | — | |
| SiliconMind-V1Role=Fine-tuned, Base Model=Qwen3-4B-Thinking, SFT=true, RL=false2026.06 | 82 | — | — | — | — | — | 89.6 | 91 | |
| SiliconMind-V1Role=Fine-tuned, Base Model=Qwen3-8B, SFT=true, RL=false2026.06 | 81 | — | — | — | — | — | 89.8 | 92.4 | |
| STGRole=Fine-tuned, Base Model=Qwen3-8B, SFT=true, RL=false2026.06 | 80.2 | — | — | — | — | — | 89.9 | 92 | |
| ChipMATE-Agents-9BType=ChipMATE (Ours), Size=9B2026.05 | 80.1 | — | — | — | — | — | 82.4 | — | |
| STGRole=Fine-tuned, Base Model=Qwen3-4B-Thinking, SFT=true, RL=false2026.06 | 80 | — | — | — | — | — | 90.2 | 91.5 | |
| DeepSeek R1Type=Foundation Models, Size=671B2026.05 | 77.5 | — | — | — | — | — | 84.7 | — | |
| ChipMATE-Verilog-9BType=ChipMATE (Ours), Size=9B2026.05 | 75.3 | — | — | — | — | — | 77.6 | — | |
| ChipMATE-Agents-4BType=ChipMATE (Ours), Size=4B2026.05 | 75 | — | — | — | — | — | 76.3 | — | |
| SiliconMind-V1Role=Fine-tuned, Base Model=Qwen2.5-C-7B-Instruct, SFT=true, RL=false2026.06 | 73.9 | — | — | — | — | — | 83.6 | 85.8 | |
| CodeV-R1Role=Fine-tuned, Base Model=Qwen2.5-C-7B-Instruct, SFT=true, RL=true2026.06 | 73.2 | — | — | — | — | — | 83.6 | 86.6 | |
| STGRole=Fine-tuned, Base Model=Qwen2.5-C-7B-Instruct, SFT=true, RL=false2026.06 | 70.5 | — | — | — | — | — | 84.9 | 89.4 | |
| CodeV-R1Type=Specialized Models, Size=7B2026.05 | 68.8 | — | — | — | — | — | 78.2 | — | |
| DeepSeek CoderType=Foundation Models, Size=236B2026.05 | 68.5 | — | — | — | — | — | 80.8 | — | |
| ChipMATE-Verilog-4BType=ChipMATE (Ours), Size=4B2026.05 | 67.4 | — | — | — | — | — | 71.8 | — | |
| DeepSeek V4Type=Foundation Models, Size=1.6T2026.05 | 67.3 | — | — | — | — | — | 80.1 | — | |
| VeriRL (paper)Role=Fine-tuned, Base Model=Qwen2.5-C-7B-Instruct, SFT=true, RL=true2026.06 | 67.2 | — | — | — | — | — | 76.1 | — | |
| CodeV-R1 (distill)Type=Specialized Models, Size=7B2026.05 | 65.2 | — | — | — | — | — | 75.2 | — | |
| GPT-4oType=Foundation Models2026.05 | 64.1 | — | — | — | — | — | 73.7 | — | |
| VeriRL (reproduced)Role=Fine-tuned, Base Model=Qwen2.5-C-7B-Instruct, SFT=true, RL=true2026.06 | 60.8 | — | — | — | — | — | 74.8 | 78.8 | |
| Qwen3-8BRole=Base, SFT=null, RL=null2026.06 | 52.5 | — | — | — | — | — | 65.4 | 69.1 | |
| Qwen3.5-9BType=Base Models, Size=9B2026.05 | 48.5 | — | — | — | — | — | 66.6 | — | |
| Qwen3.5-4BType=Base Models, Size=4B2026.05 | 41.7 | — | — | — | — | — | 60.9 | — | |
| Qwen2.5-C-7B-InstructRole=Base, SFT=null, RL=null2026.06 | 33.6 | — | — | — | — | — | 53.7 | 60.1 | |
| Qwen3-4B-ThinkingRole=Base, SFT=null, RL=null2026.06 | 21.4 | — | — | — | — | — | 30.4 | 33.4 | |
| DeepSeek-R1-FP42026.01 | — | 91 | 94.2 | 94.9 | 94.9 | 95.5 | — | — | |
| DeepSeek-R1-FP4-MCTSSTG=false, Search Strategy=Monte Carlo Tree Search2026.01 | — | 94.2 | 96.8 | 96.8 | 97.4 | 98.1 | — | — | |
| Gemini-2.0-FlashAPI Access=true2026.01 | — | 72.4 | 74.4 | 77.6 | 78.8 | 80.1 | — | — | |
| Gemini-2.0-Flash-IGRSTG=false, Search Strategy=Idea-Guided Refinement, API Access=true2026.01 | — | 76.3 | 85.3 | 86.5 | 86.5 | 86.5 | — | — | |
| Gemini-2.0-Flash-MCTSSTG=false, Search Strategy=Monte Carlo Tree Search, API Access=true2026.01 | — | 77.6 | 81.4 | 84.6 | 86.5 | 87.8 | — | — | |
| GPT-OSS-120B2026.01 | — | 92.9 | 93.6 | 94.2 | 94.2 | 94.2 | — | — | |
| GPT-OSS-120B-IGRSTG=false, Search Strategy=Idea-Guided Refinement2026.01 | — | 89.1 | 95.5 | 96.2 | 96.8 | 96.8 | — | — | |
| GPT-OSS-120B-MCTSSTG=false, Search Strategy=Monte Carlo Tree Search2026.01 | — | 89.7 | 93.6 | 94.9 | 95.5 | 96.2 | — | — | |
| REvolution (Deepseek-V3)API Access=true2026.01 | — | — | — | — | 95.5 | — | — | — | |
| REvolution (Llama-3.3-70B)API Access=true2026.01 | — | — | — | — | 88.5 | — | — | — | |
| Siliconmind-7B2026.01 | — | 82.1 | 87.8 | 89.7 | 91.7 | 91.7 | — | — | |
| Siliconmind-7B-MCTSSTG=false, Search Strategy=Monte Carlo Tree Search2026.01 | — | 84 | 85.3 | 86.5 | 90.4 | 92.3 | — | — | |
| VerilogCoder (GPT-4-Turbo)API Access=true2026.01 | — | — | — | 94.2 | — | — | — | — | |
| VerilogCoder (Llama-3-70B)2026.01 | — | — | — | 67.3 | — | — | — | — |