Code Generation on MBPP-ET
91.8Pass@1AgentCoder
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AgentCoderBackbone=GPT-42023.12 | 91.8 | 75.9 | |
| AgentCoderBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 91.8 | — | |
| AgentCoderBackbone=GPT-4-turbo2023.12 | 91.4 | 92.4 | |
| AgentCoderBackbone=GPT-4-Turbo2024.12 | 91.4 | — | |
| LCPBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 89.7 | — | |
| AgentCoderBackbone=GPT-3.5-turbo2023.12 | 89.1 | 142.1 | |
| AgentCoderBackbone=GPT-3.5-Turbo2024.12 | 89.1 | — | |
| AgentCoderBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 89.1 | — | |
| LCPBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 86.4 | — | |
| ThinkCoderBackbone=GPT-4-Turbo2024.12 | 80.5 | — | |
| ThinkCoder (GPT-4-Turbo)n=5, k=5, temperature=0.52024.12 | 80 | — | |
| ThinkCoder (GPT-4-Turbo)n=2, k=5, temperature=0.52024.12 | 79.9 | — | |
| ThinkCoder (GPT-4o)n=1, k=5, temperature=0.52024.12 | 79.9 | — | |
| ThinkCoderBackbone=GPT-4o2024.12 | 79.2 | — | |
| ThinkCoder (GPT-4o)n=2, k=5, temperature=0.52024.12 | 79.2 | — | |
| ThinkCoder (GPT-4o)n=5, k=5, temperature=0.52024.12 | 79.2 | — | |
| ThinkCoderBackbone=GPT-3.5-Turbo2024.12 | 78.8 | — | |
| ThinkCoder (GPT-4-Turbo)n=1, k=5, temperature=0.52024.12 | 78.2 | — | |
| GPT-4on=0, temperature=02024.12 | 78.2 | — | |
| AgentCoderBackbone=PaLM Coder2023.12 | 75.5 | 177.6 | |
| AgentCoderBackbone=Claude-instant-12023.12 | 75.1 | 277.4 | |
| ThinkCoder (CodeQwen1.5-7B-Chat)n=2, k=5, temperature=0.52024.12 | 75 | — | |
| ThinkCoder (CodeQwen1.5-7B-Chat)n=5, k=5, temperature=0.52024.12 | 75 | — | |
| ThinkCoder (Kimi)n=5, k=5, temperature=0.52024.12 | 74.8 | — | |
| ThinkCoder (CodeQwen1.5-7B-Chat)n=1, k=5, temperature=0.52024.12 | 74.7 | — | |
| ThinkCoder (Kimi)n=2, k=5, temperature=0.52024.12 | 74.5 | — | |
| GPT-4-Turbon=0, temperature=02024.12 | 74.4 | — | |
| ThinkCoder (Kimi)n=1, k=5, temperature=0.52024.12 | 73.5 | — | |
| CodeQwen1.5-7B-Chatn=0, temperature=02024.12 | 67.8 | — | |
| Kimin=0, temperature=02024.12 | 67.4 | — | |
| LPWBackbone=GPT-4o2024.12 | 65.8 | — | |
| LDBBackbone=GPT-4o2024.12 | 65.4 | — | |
| CodeCoRBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 65.2 | — | |
| CodeCoTBackbone=GPT-3.5-turbo2023.12 | 63 | 71.2 | |
| Self-CollaborationBackbone=GPT-42023.12 | 62.1 | 19 | |
| Self-CollaborationBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 62.1 | — | |
| FlowGenscrum+TestLLM=GPT-3.5, Process Model=Scrum, Incorporates=CodeT2024.03 | 58.7 | — | |
| MapCoderLLM=GPT-42024.05 | 57.7 | — | |
| MapCoderBackbone=GPT-4-Turbo2024.12 | 57.7 | — | |
| CodeTLLM=GPT-3.52024.03 | 56.7 | — | |
| FlowGenscrumLLM=GPT-3.5, Process Model=Scrum2024.03 | 56.7 | — | |
| CoTLLM=GPT-42024.05 | 56.2 | — | |
| DirectLLM=GPT-42024.05 | 54.7 | — | |
| MapCoderLLM=ChatGPT2024.05 | 54.4 | — | |
| MapCoderBackbone=GPT-3.5-Turbo2024.12 | 54.4 | — | |
| MapCoderBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 54.4 | — | |
| Self-debuggingBackbone=GPT-3.5-turbo2023.12 | 52.3 | 42.1 | |
| GPT-4Backbone=GPT-4, Setting=Zero-Shot2023.12 | 52.2 | — | |
| GPT-4Evaluation Strategy=Zero-Shot2026.03 | 52.2 | — | |
| ReflexionLLM=GPT-3.52024.03 | 52 | — | |
| ReflexionLLM=GPT-42024.05 | 51.9 | — | |
| Self-PlanningLLM=GPT-42024.05 | 50.4 | — | |
| Self-collaborationLLM=ChatGPT2024.05 | 49.5 | — | |
| Self-CollaborationBackbone=GPT-3.5-turbo2023.12 | 49.5 | 34.5 | |
| Few-ShotBackbone=GPT-3.5-turbo2023.12 | 48.3 | 31.2 | |
| Few-ShotBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 48.3 | — | |
| GPT-4-turboBackbone=GPT-4-turbo, Setting=Zero-Shot2023.12 | 47.5 | — | |
| ReflexionBackbone=GPT-3.5-turbo2023.12 | 47.5 | 29.1 | |
| GPT-4-turboEvaluation Strategy=Zero-Shot2026.03 | 47.5 | — | |
| ReflexionBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 47.5 | — | |
| ReflexionLLM=ChatGPT2024.05 | 47.4 | — | |
| INTERVENORBackbone=GPT-3.5-turbo2023.12 | 47.1 | 28 | |
| RAPBackbone=GPT-3.5-turbo2023.12 | 46.7 | 26.9 | |
| AnalogicalLLM=ChatGPT2024.05 | 46.1 | — | |
| ReActBackbone=GPT-3.5-turbo2023.12 | 45.9 | 24.7 | |
| Self-EditBackbone=GPT-3.5-turbo2023.12 | 45.9 | 24.7 | |
| ReActBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 45.9 | — | |
| CodeX+CodeTBackbone=CodeX (175B), Setting=Augmented2023.12 | 45.1 | — | |
| Self-PlanningLLM=ChatGPT2024.05 | 41.9 | — | |
| Self-PlaningBackbone=GPT-3.5-turbo2023.12 | 41.5 | 12.8 | |
| ToTBackbone=GPT-3.5-turbo2023.12 | 40.8 | 10.9 | |
| AnalogicalLLM=GPT-42024.05 | 40.3 | — | |
| CoTLLM=ChatGPT2024.05 | 39.6 | — | |
| CodeXBackbone=CodeX (175B), Setting=Zero-Shot2023.12 | 38.8 | — | |
| DirectLLM=ChatGPT2024.05 | 37.7 | — | |
| GPT-3.5-turboBackbone=GPT-3.5-turbo, Setting=Zero-Shot2023.12 | 36.8 | — | |
| GPT-3.5-turboEvaluation Strategy=Zero-Shot2026.03 | 36.8 | — | |
| CoTBackbone=GPT-3.5-turbo2023.12 | 34.8 | -5.4 | |
| CoTBase Model=GPT-3.5-turbo, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 34.8 | — | |
| StarCoderBackbone=StarCoder (15.5B), Setting=Zero-Shot2023.12 | 33.4 | — | |
| CodeGen-MonoBackbone=CodeGen-Mono (16.1B), Setting=Zero-Shot2023.12 | 31.6 | — | |
| ThinkCoder (LLama2-7B-Chat)n=5, k=5, temperature=0.52024.12 | 29.8 | — | |
| ThinkCoder (LLama2-7B-Chat)n=2, k=5, temperature=0.52024.12 | 29.5 | — | |
| ThinkCoder (LLama2-7B-Chat)n=1, k=5, temperature=0.52024.12 | 29 | — | |
| PaLM CoderBackbone=PaLM Coder, Setting=Zero-Shot2023.12 | 27.2 | — | |
| LLama2-7B-Chatn=0, temperature=02024.12 | 22.9 | — | |
| CodeGeeXBackbone=CodeGeeX (13B), Setting=Zero-Shot2023.12 | 20.4 | — | |
| Claude-instant-1Backbone=Claude-instant-1, Setting=Zero-Shot2023.12 | 19.9 | — | |
| Claude-instant-1Evaluation Strategy=Zero-Shot2026.03 | 19.9 | — | |
| IncoderBackbone=Incoder (6.7B), Setting=Zero-Shot2023.12 | 14.3 | — | |
| IncoderEvaluation Strategy=Zero-Shot, Model Parameters=6.7B2026.03 | 14.3 | — |