Code Generation on Aider-Polyglot
61.7Pass@1o1
Evaluation results
| Method | Links | |
|---|---|---|
| o12026.03 | 61.7 | |
| MTL (I)Model Backbone=DeepSeek V3.22026.04 | 44.7 | |
| ZeroShotModel Backbone=DeepSeek V3.22026.04 | 43.3 | |
| MTL (S)Model Backbone=GPT-5-mini2026.04 | 37 | |
| MTL (I)Model Backbone=Qwen3-Coder-480B-A35B-Instruct2026.04 | 36 | |
| MTL (T)Model Backbone=GPT-5-mini2026.04 | 35.7 | |
| ZeroShotModel Backbone=Qwen3-Coder-480B-A35B-Instruct2026.04 | 35.7 | |
| MTL (W)Model Backbone=GPT-5-mini2026.04 | 35 | |
| MTL (I)Model Backbone=GPT-5-mini2026.04 | 34.7 | |
| ZeroShotModel Backbone=GPT-5-mini2026.04 | 34.3 | |
| EVOLVECODER-4B (r3)Refinement round=r3, Backbone=Qwen3-4B (Thinking)2026.03 | 27.4 | |
| EVOLVECODER-4B (r2)Refinement round=r2, Backbone=Qwen3-4B (Thinking)2026.03 | 27.1 | |
| Critique-Coder-4B2026.03 | 24.4 | |
| EVOLVECODER-4B (r0)Refinement round=r0, Backbone=Qwen3-4B (Thinking)2026.03 | 24.4 | |
| EVOLVECODER-4B (r1)Refinement round=r1, Backbone=Qwen3-4B (Thinking)2026.03 | 24.4 | |
| Qwen3-4B (Thinking)Thinking-mode sampling=true2026.03 | 21.8 | |
| DeepSeek-R1-Distill-14B2026.03 | 18.6 | |
| DeepCoder-14B2026.03 | 18.4 | |
| DeepSeek-V2.5-238B2026.03 | 17.8 |