Failure Diagnosis on MBPP
50Macro Similarity ScorePROBELLM
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| PROBELLMTarget Model=GPT-4.1, Model Family=OpenAI Family, Release Date=Apr 20252026.02 | 50 | 50 | 22 | 36 | 6 | |
| PROBELLMTarget Model=GPT-5.2, Model Family=OpenAI Family, Release Date=Dec 20252026.02 | 35 | 65 | 25 | 29 | 5 | |
| PROBELLMTarget Model=Ministral-8B, Model Family=Mistral Family, Release Date=Oct 20242026.02 | 32 | 68 | 77 | 74 | 19 | |
| PROBELLMTarget Model=Mistral-7B-Ins.-v0.1, Model Family=Mistral Family, Release Date=Sep 20232026.02 | 31 | 69 | 82 | 81 | 16 | |
| PROBELLMTarget Model=Ministral-3-8B-2512, Model Family=Mistral Family, Release Date=Dec 20252026.02 | 31 | 69 | 49 | 57 | 16 | |
| PROBELLMTarget Model=GPT-3.5-turbo, Model Family=OpenAI Family, Release Date=Mar 20232026.02 | 26 | 74 | 80 | 75 | 16 | |
| PROBELLMTarget Model=GPT-4o-mini, Model Family=OpenAI Family, Release Date=Jul 20242026.02 | 20 | 80 | 65 | 63 | 10 | |
| PROBELLMTarget Model=Mistral-7B-Ins., Model Family=Mistral Family, Release Date=May 20242026.02 | 13 | 87 | 87 | 78 | 20 |