Code Completion on CVDP non-agentic 1.0
24.47Pass@1GPT-o4 Mini
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-o4 Minimode=agentic framework2025.12 | 24.47 | |
| GPT-o4 Minimode=single-shot2025.12 | 17.02 | |
| Granite-4mode=single-shot2025.12 | 9.57 | |
| Nemotron-Minimode=single-shot2025.12 | 4.26 | |
| SmolLMmode=single-shot2025.12 | 1.03 | |
| SmolLMmode=agentic framework2025.12 | 1.03 | |
| Nemotron-Minimode=agentic framework2025.12 | 0 | |
| DeepSeek-R1mode=single-shot2025.12 | 0 | |
| DeepSeek-R1mode=agentic framework2025.12 | 0 | |
| Granite-4mode=agentic framework2025.12 | 0 |