Code Generation on MBPP (MA, MI, Error Rate)
0.68MAPROBELLM
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| PROBELLMTarget Model=GPT-oss-20b2026.02 | 0.68 | 0.32 | 0.32 | |
| PROBELLMTarget Model=Deepseek-v3.22026.02 | 0.65 | 0.35 | 0.46 | |
| PROBELLMTarget Model=olmo-3-7b-ins.2026.02 | 0.59 | 0.41 | 0.76 | |
| PROBELLMTarget Model=Gemini-2.5-flash2026.02 | 0.53 | 0.47 | 0.54 | |
| PROBELLMTarget Model=granite-4.02026.02 | 0.5 | 0.5 | 0.83 | |
| PROBELLMTarget Model=Claude-3.5-sonnet2026.02 | 0.42 | 0.58 | 0.62 | |
| PROBELLMTarget Model=ministral-14b2026.02 | 0.42 | 0.58 | 0.64 | |
| PROBELLMTarget Model=Llama-3.1-8b-ins.2026.02 | 0.4 | 0.6 | 0.9 | |
| PROBELLMTarget Model=phi-42026.02 | 0.39 | 0.61 | 0.78 | |
| PROBELLMTarget Model=Grok-4.1-fast2026.02 | 0.35 | 0.65 | 0.39 | |
| PROBELLMTarget Model=GPT-4o-mini2026.02 | 0.31 | 0.69 | 0.76 | |
| PROBELLMTarget Model=devstral2026.02 | 0.27 | 0.73 | 0.5 |