Diffusion Acceleration Code Generation on DiffBench (test)
90.24Level 1 AccuracyClaude Sonnet 4 w/ DiffAgent
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Claude Sonnet 4 w/ DiffAgentBase Model=Claude Sonnet 4, Agent Framework=DiffAgent, Evaluation Protocol=augmented2026.01 | 90.24 | 91.38 | 99.23 | 33.33 | 63.44 | 81.59 | |
| Claude Sonnet 4Base Model=Claude Sonnet 4, Agent Framework=None, Evaluation Protocol=out-of-the-box2026.01 | 78.04 | 72.41 | 76.25 | 5.38 | 8.6 | 54.3 | |
| o3-mini w/ DiffAgentBase Model=o3-mini, Agent Framework=DiffAgent, Evaluation Protocol=augmented2026.01 | 73.17 | 70.69 | 69.73 | 22.58 | 27.96 | 56.46 | |
| GPT-4.1Base Model=GPT-4.1, Agent Framework=None, Evaluation Protocol=out-of-the-box2026.01 | 56.1 | 18.97 | 7.28 | 10.75 | 12.9 | 14.24 | |
| o3-miniBase Model=o3-mini, Agent Framework=None, Evaluation Protocol=out-of-the-box2026.01 | 41.46 | 24.14 | 4.6 | 9.68 | 6.45 | 11.92 | |
| Gemini 2.5 FlashBase Model=Gemini 2.5 Flash, Agent Framework=None, Evaluation Protocol=out-of-the-box2026.01 | 39.02 | 29.31 | 7.66 | 2.15 | 1.08 | 12.09 |