Code Generation on MBPP Original
70.86Pass RateClaude 3.7 Sonnet
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Claude 3.7 SonnetProcessing=Postprocessed2026.01 | 70.86 | 29.14 | 0 | |
| o4-miniProcessing=Postprocessed2026.01 | 70.05 | 29.68 | 0.27 | |
| Gemini-2.0-FlashProcessing=Postprocessed2026.01 | 68.72 | 31.02 | 0.26 | |
| o4-miniProcessing=Filtered2026.01 | 66.58 | 33.42 | 0 | |
| GPT-4.1-MiniProcessing=Postprocessed2026.01 | 66.04 | 30.48 | 0 | |
| Gemini-2.0-FlashProcessing=Filtered2026.01 | 64.97 | 35.03 | 0 | |
| Claude 3.7 SonnetProcessing=Filtered2026.01 | 63.37 | 36.63 | 0 | |
| Qwen2.5-7b-InstructProcessing=Postprocessed2026.01 | 60.43 | 37.7 | 1.87 | |
| GPT-4.1-MiniProcessing=Filtered2026.01 | 58.02 | 36.1 | 5.88 | |
| Qwen2.5-7b-InstructProcessing=Filtered2026.01 | 52.67 | 45.99 | 1.34 | |
| Qwen2.5-3b-CoderProcessing=Postprocessed2026.01 | 52.41 | 42.51 | 5.08 | |
| Qwen2.5-3b-CoderProcessing=Filtered2026.01 | 46.79 | 48.93 | 4.28 |