Code Generation on MBPP New
70.13Pass Rateo4-mini
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| o4-miniProcessing=Postprocessed2026.01 | 70.13 | 29.68 | 0.19 | |
| GPT-4.1-MiniProcessing=Postprocessed2026.01 | 67.35 | 29.68 | 2.97 | |
| Claude 3.7 SonnetProcessing=Postprocessed2026.01 | 64.75 | 35.25 | 0 | |
| Gemini-2.0-FlashProcessing=Postprocessed2026.01 | 63.64 | 36.18 | 0.19 | |
| o4-miniProcessing=Filtered2026.01 | 58.26 | 41.19 | 0.56 | |
| GPT-4.1-MiniProcessing=Filtered2026.01 | 56.96 | 40.63 | 2.41 | |
| Claude 3.7 SonnetProcessing=Filtered2026.01 | 55.29 | 44.71 | 0 | |
| Gemini-2.0-FlashProcessing=Filtered2026.01 | 53.99 | 45.64 | 0.37 | |
| Qwen2.5-7b-InstructProcessing=Postprocessed2026.01 | 45.83 | 51.58 | 2.6 | |
| Qwen2.5-3b-CoderProcessing=Postprocessed2026.01 | 38.78 | 49.54 | 11.69 | |
| Qwen2.5-7b-InstructProcessing=Filtered2026.01 | 34.32 | 61.6 | 4.08 | |
| Qwen2.5-3b-CoderProcessing=Filtered2026.01 | 31.35 | 55.47 | 13.17 |