Hardware Verification Testbench Generation on CVDP-ECov
69.9Coverage Pass RateQwen3-235b-A22b-2507
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3-235b-A22b-2507Execution Mode=Agentic, Model Type=General Purpose2026.02 | 69.9 | 83.2 | |
| LLM4CovExecution Mode=Agentic, Model Type=LLM4Cov, Training Stage=Stage22026.02 | 69.2 | 90.4 | |
| LLM4CovExecution Mode=Agentic, Model Type=LLM4Cov, Training Stage=Stage12026.02 | 67 | 88.2 | |
| Qwen3-Coder-30B-A3BExecution Mode=Agentic, Model Type=Coding Specialized2026.02 | 63.9 | 79.9 | |
| llama-4-maverick (400B)Execution Mode=Agentic, Model Type=General Purpose2026.02 | 60.2 | 81.7 | |
| LLM4CovExecution Mode=Agentic, Model Type=LLM4Cov, Training Stage=Stage02026.02 | 60.2 | 82.1 | |
| Qwen2.5-Coder-32BExecution Mode=Agentic, Model Type=Coding Specialized2026.02 | 59.3 | 79.6 | |
| Qwen3-30B-A3B-2507Execution Mode=Agentic, Model Type=General Purpose2026.02 | 50.6 | 68 | |
| LLM4CovExecution Mode=Direct Inference, Model Type=LLM4Cov, Training Stage=Stage22026.02 | 50.6 | 76.4 | |
| LLM4CovExecution Mode=Direct Inference, Model Type=LLM4Cov, Training Stage=Stage12026.02 | 48.2 | 77.2 | |
| LLM4CovExecution Mode=Direct Inference, Model Type=LLM4Cov, Training Stage=Stage02026.02 | 45.5 | 74.9 | |
| Qwen2.5-72BExecution Mode=Agentic, Model Type=General Purpose2026.02 | 42.9 | 64.1 | |
| Qwen3-30B-A3B-2507Execution Mode=Direct Inference, Model Type=General Purpose2026.02 | 30.1 | 52.2 | |
| Qwen3-235b-A22b-2507Execution Mode=Direct Inference, Model Type=General Purpose2026.02 | 29.9 | 47 | |
| CodeV-R1-RL-Qwen-7BExecution Mode=Agentic, Model Type=Hardware Specific2026.02 | 29.9 | 55.5 | |
| Qwen3-Coder-30B-A3BExecution Mode=Direct Inference, Model Type=Coding Specialized2026.02 | 28.4 | 48.6 | |
| Qwen3-4B-2507 (Base)Execution Mode=Agentic, Model Type=LLM4Cov, Training Stage=Base2026.02 | 28.4 | 48.5 | |
| VeriCoder-Qwen2.5-14BExecution Mode=Agentic, Model Type=Hardware Specific2026.02 | 28.2 | 47.9 | |
| Qwen2.5-Coder-32BExecution Mode=Direct Inference, Model Type=Coding Specialized2026.02 | 23.4 | 52.1 | |
| llama-4-maverick (400B)Execution Mode=Direct Inference, Model Type=General Purpose2026.02 | 23.1 | 47.3 | |
| Qwen2.5-Coder-7BExecution Mode=Agentic, Model Type=Coding Specialized2026.02 | 20.5 | 34.4 | |
| VeriCoder-Qwen2.5-14BExecution Mode=Direct Inference, Model Type=Hardware Specific2026.02 | 17.1 | 37.3 | |
| Qwen2.5-72BExecution Mode=Direct Inference, Model Type=General Purpose2026.02 | 16.4 | 32.2 | |
| VeriReason-Qwen2.5-7BExecution Mode=Agentic, Model Type=Hardware Specific2026.02 | 15.9 | 26.4 | |
| Qwen3-4B-2507 (Base)Execution Mode=Direct Inference, Model Type=LLM4Cov, Training Stage=Base2026.02 | 14.9 | 23.3 | |
| CodeV-R1-RL-Qwen-7BExecution Mode=Direct Inference, Model Type=Hardware Specific2026.02 | 14.5 | 32.5 | |
| Qwen2.5-Coder-7BExecution Mode=Direct Inference, Model Type=Coding Specialized2026.02 | 11.3 | 26.6 | |
| VeriReason-Qwen2.5-7BExecution Mode=Direct Inference, Model Type=Hardware Specific2026.02 | 8.9 | 16.2 |