Code Generation on MBPP-Hard
62.56Pass Rateo4-mini
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| o4-miniProcessing=Postprocessed2026.01 | 62.56 | 33.33 | 4.11 | — | — | |
| Claude 3.7 SonnetProcessing=Postprocessed2026.01 | 56.62 | 40.64 | 2.74 | — | — | |
| GPT-4.1-MiniProcessing=Postprocessed2026.01 | 55.25 | 38.81 | 5.94 | — | — | |
| Gemini-2.0-FlashProcessing=Postprocessed2026.01 | 49.32 | 45.66 | 5.02 | — | — | |
| o4-miniProcessing=Filtered2026.01 | 47.75 | 49.55 | 2.7 | — | — | |
| Claude 3.7 SonnetProcessing=Filtered2026.01 | 45.05 | 52.25 | 2.7 | — | — | |
| Gemini-2.0-FlashProcessing=Filtered2026.01 | 44.59 | 52.25 | 3.15 | — | — | |
| GPT-4.1-MiniProcessing=Filtered2026.01 | 44.14 | 50.45 | 5.41 | — | — | |
| Qwen2.5-7b-InstructProcessing=Postprocessed2026.01 | 30.14 | 63.01 | 6.85 | — | — | |
| Qwen2.5-3b-CoderProcessing=Postprocessed2026.01 | 24.2 | 53.88 | 21.92 | — | — | |
| Qwen2.5-3b-CoderProcessing=Filtered2026.01 | 21.62 | 54.95 | 23.42 | — | — | |
| Qwen2.5-7b-InstructProcessing=Filtered2026.01 | 19.82 | 72.52 | 7.66 | — | — | |
| TextBFGSKnowledge Base=None2026.01 | — | — | — | 85.47 | 48.72 | |
| TextBFGSRetrieval=Gradient-based, Knowledge Base=MBPP KB2026.01 | — | — | — | 94.02 | 74.36 | |
| TextBFGS-REMORetrieval=Input-based, Knowledge Base=MBPP KB2026.01 | — | — | — | 95.73 | 78.63 | |
| TextGrad2026.01 | — | — | — | 85.47 | 48.72 | |
| TextGrad-Momentum2026.01 | — | — | — | 88.89 | 58.12 |