Code Generation on 133 PowerShell CodeGen prompts
46FRate (%)o3-mini
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| o3-miniDecoding=Default (no temp control)2026.01 | 46 | 30 | |
| GPT-4oDecoding=Deterministic (temp=0)2026.01 | 42 | 34 | |
| Qwen2.5-7BDecoding=Deterministic (temp=0)2026.01 | 24 | 34 | |
| Qwen2.5-Coder-7BDecoding=Deterministic (temp=0)2026.01 | 20 | 22 | |
| DeepSeek-R1-Distill-Qwen-7BDecoding=Deterministic (temp=0)2026.01 | 2 | 0 |