Code Comprehension on CVDP (Code Verilog Design Problems) (test)
106Total Passesphi3.5-mini-instruct
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| phi3.5-mini-instructtotal_problems=1862025.12 | 106 | 56.99 | 47.06 | 37.93 | 82.35 | 92.31 | |
| deepseek-r1total_problems=1862025.12 | 106 | 56.99 | 50 | 37.93 | 76 | 92 | |
| gpt-osstotal_problems=1862025.12 | 94 | 50.54 | 58.82 | 37.93 | 47 | 62 | |
| GPT-4ototal_problems=1862025.12 | 74 | 39.78 | 33.33 | 11.11 | 91.18 | 96.15 | |
| nemotron-minitotal_problems=1862025.12 | 62 | 33.33 | 29.41 | 10.34 | 64.71 | 53.85 | |
| GPT-3.5-turbototal_problems=1862025.12 | 61 | 32.8 | 18.18 | 0 | 82.35 | 96.15 | |
| GPT-o4 Minitotal_problems=1862025.12 | 61 | 32.8 | 13.64 | 6 | 82.35 | 96.15 | |
| smollm2total_problems=1862025.12 | 14 | 7.53 | 11.76 | 10.34 | 0 | 0 |