List your company
ResearchFormerly WizWand

Code Generation on MBPP (pass@1 accuracy)

94.2Pass@1 AccuracyGemini-2.5-Pro
14.53635.21855.976.582Aug 8, 2024Nov 11, 2024Feb 15, 2025May 22, 2025Aug 26, 2025Nov 30, 2025Mar 6, 2026
Updated 7mo ago

Evaluation results

MethodLinks
Gemini-2.5-ProInstitution=Google2026.0394.2
LCPBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.0392.51
AgentCoderBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.0391.8
LCPBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0391.1
AgentCoderBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0389.9
MetaGPTBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.0387.7
GPT-4.1Institution=OpenAI2026.0385.2
OneFlowModel=Claude 3.5 Haiku2026.0184.4
GPT-5.1Institution=OpenAI2026.0384
AFlowModel=Claude 3.5 Haiku2026.0183.6
ReflexiCoder-8B (Multiple)Institution=Ours, setup=full iterative reasoning-reflection setup2026.0382
ReflexiCoder-8B (Single)Institution=Ours, setup=single-attempt without system prompt2026.0381.8
Self-DebuggingBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.0380.6
CodeCoRBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0379.2
MapCoderBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0378.9
Self-CollaborationBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.0378.9
ReflexionBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.0377.1
CoIPOModel=Qwen2.5-72B2026.0277
Claude-Sonnet-4.5Institution=Anthropic2026.0376.8
Qwen2.5-Coder-7B-InstructInstitution=Alibaba2026.0375.8
IOModel=Claude 3.5 Haiku2026.0174.5
BaseModel=Qwen2.5-72B2026.0274.2
Qwen3-8BInstitution=Alibaba2026.0370.2
ReflexionBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0370
CodeLlamaEvaluation Strategy=Zero-Shot, Model Parameters=34B2026.0369.3
Seed-Coder-8B-InstructInstitution=ByteDance2026.0368.4
GPT-4Evaluation Strategy=Zero-Shot2026.0368.3
ReActBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0367
Few-ShotBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0365.8
DeepSeek-Coder-7B-InstructInstitution=DeepSeek2026.0365
GPT-4-turboEvaluation Strategy=Zero-Shot2026.0363.4
DeepCoder-14B-PreviewInstitution=rLLM2026.0363.4
Ledex-RL-13BInstitution=Purdue2026.0361.98
Ledex-RL-7BInstitution=Purdue2026.0357.92
CodeGemma-7B-ITInstitution=Google2026.0353.2
GPT-3.5-turboEvaluation Strategy=Zero-Shot2026.0352.2
CoTBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.0346.1
CoIPOModel=Qwen2.5-14B2026.0243.2
CodeLlama-7b-InstructInstitution=Meta2026.0338.21
BA-LoRAPEFT Method=BA-LoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0836.86
BaseModel=Qwen2.5-14B2026.0236.4
DeepCoder-1.5B-PreviewInstitution=rLLM2026.0336
BaseModel=Qwen2.5-7B2026.0235.6
CoIPOModel=Qwen2.5-7B2026.0232.8
Claude-instant-1Evaluation Strategy=Zero-Shot2026.0326.9
CorDA++PEFT Method=CorDA++, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0824.74
CorDAPEFT Method=CorDA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0824.15
PiSSAPEFT Method=PiSSA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0823.84
LoRA+PEFT Method=LoRA+, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0823.69
Full FTPEFT Method=Full Fine-tuning, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0823.64
LoRA-GAPEFT Method=LoRA-GA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0823.05
DoRAPEFT Method=DoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0821.6
LoRAPEFT Method=LoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0821.51
MiLoRAPEFT Method=MiLoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0820.22
AdaLoRAPEFT Method=AdaLoRA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0820.19
LoRA-FAPEFT Method=LoRA-FA, Base Model=LLaMA-2-7B, Fine-tuning Dataset=CodeFeedback2024.0820.01
BaseModel=Llama-7B2026.0218
CoIPOModel=Llama-7B2026.0217.8
IncoderEvaluation Strategy=Zero-Shot, Model Parameters=6.7B2026.0317.6