Python Coding on MBPP standard (test)
85.25Pass@1 AccuracyCoT
Evaluation Results
| Method | Links | |
|---|---|---|
| CoTModel=Qwen-3-8B, Prompting Strategy=Chain-of-Thought2026.02 | 85.25 | |
| Ann BrownModel=Qwen-3-8B, Prompting Strategy=Ann Brown Prompting2026.02 | 79.39 | |
| StdModel=Qwen-3-8B, Prompting Strategy=Standard2026.02 | 74.94 | |
| MPModel=Qwen-3-8B, Prompting Strategy=Metacognitive Prompting2026.02 | 73.07 | |
| CoTModel=Llama-3-8B, Prompting Strategy=Chain-of-Thought2026.02 | 69.79 | |
| MPModel=Llama-3-8B, Prompting Strategy=Metacognitive Prompting2026.02 | 67.92 | |
| Ann BrownModel=Llama-3-8B, Prompting Strategy=Ann Brown Prompting2026.02 | 66.51 | |
| InternLM2-Chat-20BShots=5-shot, Model Size Group=13~20B Models2024.03 | 65.8 | |
| StdModel=Llama-3-8B, Prompting Strategy=Standard2026.02 | 64.64 | |
| InternLM2-Chat-20B-SFTShots=5-shot, Model Size Group=13~20B Models2024.03 | 63.4 | |
| Mixtral-8x7B-Instruct-v0.1Shots=5-shot, Model Size Group=13~20B Models2024.03 | 59.5 | |
| ChatGLM3-6BShots=5-shot, Model Size Group=< 7B Models2024.03 | 54.9 | |
| InternLM2-Chat-7BShots=5-shot, Model Size Group=< 7B Models2024.03 | 52.5 | |
| InternLM2-Chat-7B-SFTShots=5-shot, Model Size Group=< 7B Models2024.03 | 47.5 | |
| phi-1.5-webzero-shot=true, number of parameters=1.3B2023.09 | 43.5 | |
| Baichuan2-13B-ChatShots=5-shot, Model Size Group=13~20B Models2024.03 | 40.9 | |
| Llama-65Bzero-shot=true, number of parameters=65B2023.09 | 37.7 | |
| phi-1.5zero-shot=true, number of parameters=1.3B2023.09 | 37.7 | |
| Baichuan2-7B-ChatShots=5-shot, Model Size Group=< 7B Models2024.03 | 37.7 | |
| Qwen-7B-ChatShots=5-shot, Model Size Group=< 7B Models2024.03 | 33.9 | |
| Llama2-7B-ChatShots=5-shot, Model Size Group=< 7B Models2024.03 | 30.4 | |
| Qwen-14B-ChatShots=5-shot, Model Size Group=13~20B Models2024.03 | 29.2 | |
| phi-1.5-web-onlyzero-shot=true, number of parameters=1.3B2023.09 | 27.3 | |
| Mistral-7B-Instruct-v0.2Shots=5-shot, Model Size Group=< 7B Models2024.03 | 23.7 | |
| MPT-7Bzero-shot=true, number of parameters=7B2023.09 | 22.6 | |
| Llama2-7Bzero-shot=true, number of parameters=7B2023.09 | 20.8 | |
| Llama2-13B-ChatShots=5-shot, Model Size Group=13~20B Models2024.03 | 19.8 | |
| Llama-7Bzero-shot=true, number of parameters=7B2023.09 | 17.7 | |
| Falcon 7Bzero-shot=true, number of parameters=7B2023.09 | 11.7 | |
| Falcon-rw-1.3Bzero-shot=true, number of parameters=1.3B2023.09 | 0 | |
| OPT-1.3Bzero-shot=true, number of parameters=1.3B2023.09 | 0 | |
| GPT2 XL 1.5Bzero-shot=true, number of parameters=1.5B2023.09 | 0 |