RTL – Natural Language Specification to Code on CVDP non-agentic 1.0
30.77Pass@1GPT-o4 Mini
Evaluation Results
| Method | Links | |
|---|---|---|
| GPT-o4 Minimode=single-shot2025.12 | 30.77 | |
| GPT-o4 Minimode=agentic framework2025.12 | 17.95 | |
| Granite-4mode=single-shot2025.12 | 6.41 | |
| Nemotron-Minimode=single-shot2025.12 | 1.28 | |
| Nemotron-Minimode=agentic framework2025.12 | 0 | |
| SmolLMmode=single-shot2025.12 | 0 | |
| SmolLMmode=agentic framework2025.12 | 0 | |
| DeepSeek-R1mode=single-shot2025.12 | 0 | |
| DeepSeek-R1mode=agentic framework2025.12 | 0 | |
| Granite-4mode=agentic framework2025.12 | 0 |