Code Generation on MBPP (pass@1 accuracy)
94.2Pass@1 AccuracyGemini-2.5-Pro
Evaluation Results
| Method | Links | |
|---|---|---|
| Gemini-2.5-ProInstitution=Google2026.03 | 94.2 | |
| LCPBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 92.51 | |
| AgentCoderBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 91.8 | |
| LCPBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 91.1 | |
| AgentCoderBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 89.9 | |
| MetaGPTBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 87.7 | |
| GPT-4.1Institution=OpenAI2026.03 | 85.2 | |
| OneFlowModel=Claude 3.5 Haiku2026.01 | 84.4 | |
| GPT-5.1Institution=OpenAI2026.03 | 84 | |
| AFlowModel=Claude 3.5 Haiku2026.01 | 83.6 | |
| ReflexiCoder-8B (Multiple)Institution=Ours, setup=full iterative reasoning-reflection setup2026.03 | 82 | |
| ReflexiCoder-8B (Single)Institution=Ours, setup=single-attempt without system prompt2026.03 | 81.8 | |
| Self-DebuggingBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 80.6 | |
| CodeCoRBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 79.2 | |
| MapCoderBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 78.9 | |
| Self-CollaborationBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 78.9 | |
| ReflexionBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 77.1 | |
| CoIPOModel=Qwen2.5-72B2026.02 | 77 | |
| Claude-Sonnet-4.5Institution=Anthropic2026.03 | 76.8 | |
| Qwen2.5-Coder-7B-InstructInstitution=Alibaba2026.03 | 75.8 | |
| IOModel=Claude 3.5 Haiku2026.01 | 74.5 | |
| BaseModel=Qwen2.5-72B2026.02 | 74.2 | |
| Qwen3-8BInstitution=Alibaba2026.03 | 70.2 | |
| ReflexionBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 70 | |
| CodeLlamaEvaluation Strategy=Zero-Shot, Model Parameters=34B2026.03 | 69.3 | |
| Seed-Coder-8B-InstructInstitution=ByteDance2026.03 | 68.4 | |
| GPT-4Evaluation Strategy=Zero-Shot2026.03 | 68.3 | |
| ReActBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 67 | |
| Few-ShotBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 65.8 | |
| DeepSeek-Coder-7B-InstructInstitution=DeepSeek2026.03 | 65 | |
| GPT-4-turboEvaluation Strategy=Zero-Shot2026.03 | 63.4 | |
| DeepCoder-14B-PreviewInstitution=rLLM2026.03 | 63.4 | |
| Ledex-RL-13BInstitution=Purdue2026.03 | 61.98 | |
| Ledex-RL-7BInstitution=Purdue2026.03 | 57.92 | |
| CodeGemma-7B-ITInstitution=Google2026.03 | 53.2 | |
| GPT-3.5-turboEvaluation Strategy=Zero-Shot2026.03 | 52.2 | |
| CoTBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 46.1 | |
| CoIPOModel=Qwen2.5-14B2026.02 | 43.2 | |
| CodeLlama-7b-InstructInstitution=Meta2026.03 | 38.21 | |
| BA-LoRAPEFT Method=BA-LoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 36.86 | |
| BaseModel=Qwen2.5-14B2026.02 | 36.4 | |
| DeepCoder-1.5B-PreviewInstitution=rLLM2026.03 | 36 | |
| BaseModel=Qwen2.5-7B2026.02 | 35.6 | |
| CoIPOModel=Qwen2.5-7B2026.02 | 32.8 | |
| Claude-instant-1Evaluation Strategy=Zero-Shot2026.03 | 26.9 | |
| CorDA++PEFT Method=CorDA++, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 24.74 | |
| CorDAPEFT Method=CorDA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 24.15 | |
| PiSSAPEFT Method=PiSSA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 23.84 | |
| LoRA+PEFT Method=LoRA+, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 23.69 | |
| Full FTPEFT Method=Full Fine-tuning, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 23.64 | |
| LoRA-GAPEFT Method=LoRA-GA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 23.05 | |
| DoRAPEFT Method=DoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 21.6 | |
| LoRAPEFT Method=LoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 21.51 | |
| MiLoRAPEFT Method=MiLoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 20.22 | |
| AdaLoRAPEFT Method=AdaLoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 20.19 | |
| LoRA-FAPEFT Method=LoRA-FA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.08 | 20.01 | |
| BaseModel=Llama-7B2026.02 | 18 | |
| CoIPOModel=Llama-7B2026.02 | 17.8 | |
| IncoderEvaluation Strategy=Zero-Shot, Model Parameters=6.7B2026.03 | 17.6 |