Hardware Generation on ResBench 56 problems
100Compile Success RateCodeV
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| CodeVmode=instruction-tuned2026.05 | 100 | 50 | 50 | 46.4 | 46.4 | 46.4 | 46.4 | 46.4 | 46.4 | 46.4 | |
| CKTFORMALIZER (synth)loop type=iterative compile-fix with Lean type checking, synthesis feedback=Yosys synthesis in agent loop2026.05 | 100 | 60.7 | 60.7 | 100 | 100 | 98.2 | 100 | 58.9 | 60.7 | 99.6 | |
| Direct SVmode=single LLM call2026.05 | 98.2 | 64.3 | 65.5 | 58.9 | 58.9 | 58.9 | 58.9 | 57.1 | 57.1 | 58.9 | |
| CKTFORMALIZERloop type=iterative compile-fix with Lean type checking2026.05 | 89.3 | 60.7 | 68 | 87.5 | 87.5 | 85.7 | 87.5 | 58.9 | 58.9 | 87.1 | |
| RTLCodermodel size=7B, mode=fine-tuned2026.05 | 85.7 | 51.8 | 60.4 | 48.2 | 48.2 | 46.4 | 48.2 | 48.2 | 48.2 | 47.8 |