Code Generation on HumanEval
7,927Pass@1MATRIX-Gen-SFT
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MATRIX-Gen-SFTBase model=Qwen-2.5-7B2024.10 | 7,927 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Tulu 3-SFTBase model=Qwen-2.5-7B2024.10 | 7,866 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ShareGPTBase model=Qwen-2.5-7B2024.10 | 7,439 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evol-InstructBase model=Qwen-2.5-7B2024.10 | 7,439 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WildChatBase model=Qwen-2.5-7B2024.10 | 7,378 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenHermesBase model=Qwen-2.5-7B2024.10 | 7,195 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Magpie-SFTBase model=Qwen-2.5-7B2024.10 | 7,134 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Tulu v2 MixBase model=Qwen-2.5-7B2024.10 | 6,951 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MATRIX-Gen-SFTBase model=Meta-Llama-3-8B2024.10 | 4,930 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WildChatBase model=Meta-Llama-3-8B2024.10 | 4,817 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Magpie-SFTBase model=Meta-Llama-3-8B2024.10 | 4,329 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Tulu 3-SFTBase model=Meta-Llama-3-8B2024.10 | 4,207 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evol-InstructBase model=Meta-Llama-3-8B2024.10 | 3,659 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Tulu v2 MixBase model=Meta-Llama-3-8B2024.10 | 3,659 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenHermesBase model=Meta-Llama-3-8B2024.10 | 3,354 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MASFly2026.02 | 98.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PromptBridgeSource Model=GPT-4o, Target Model=o4-mini2025.12 | 98.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LDBBackbone=GPT-4o2024.12 | 98.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LPWBackbone=GPT-4o2024.12 | 98.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gpt-oss-20b-highGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 98.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5 OptimizerSource Model=GPT-4o, Target Model=o4-mini2025.12 | 98.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude-Sonnet-4.5Institution=Anthropic2026.03 | 98.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarm2026.02 | 97.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlow2026.02 | 97.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GEPASource Model=GPT-4o, Target Model=o4-mini2025.12 | 97.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini-2.5-ProInstitution=Google2026.03 | 97.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PromptBridgeSource Model=GPT-4o, Target Model=o32025.12 | 97.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MegaAgent2026.02 | 97.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoderBackbone=GPT-4o2024.12 | 97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoder (GPT-4o)n=2, k=5, temperature=0.52024.12 | 97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoder (GPT-4o)n=5, k=5, temperature=0.52024.12 | 97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-14BGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GEPASource Model=GPT-4o, Target Model=o32025.12 | 96.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MIPROv2Source Model=GPT-4o, Target Model=o4-mini2025.12 | 96.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LCPBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 96.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentVerse2026.02 | 96.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentSquare2026.02 | 96.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct TransferSource Model=GPT-4o, Target Model=o4-mini2025.12 | 96.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4.1Institution=OpenAI2026.03 | 96.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentCoderBackbone=GPT-42023.12 | 96.3 | — | 42.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoder (GPT-4o)n=1, k=5, temperature=0.52024.12 | 96.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentCoderBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 96.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-30B-A3BGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 96.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaGPT2026.02 | 96.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TopoDIMBase model=GPT-OSS-120B, Ada.=true, DIM.=true, Dec.=true2026.01 | 95.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MURPHY - MARSBackbone=Qwen3-4B, Rollouts=72, Iterations=32025.11 | 95.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.24 | — | — | |
| ReflexiCoder-8B (Multiple)Institution=Ours, setup=full iterative reasoning-reflection setup2026.03 | 95.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReAct2026.02 | 95.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5.1Institution=OpenAI2026.03 | 95.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base2026.02 | 94.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TopoDIMBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=true, Dec.=true2026.01 | 94.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRPOBackbone=Qwen3-4B, Rollouts=72, Iterations=32025.11 | 94.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.22 | — | — | |
| DAAOLLM=LLM Pool2025.09 | 94.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 83.26 | — | — | — | — | |
| ReflexiCoder-8B (Single)Institution=Ours, setup=single-attempt without system prompt2026.03 | 94.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeCoRBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 94.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ST-EVOBackbone=gpt-oss-120b2026.02 | 94.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFA-MAS (fine-tuned)Training protocol=One-for-All Topology Design (Fine-tuned)2026.01 | 94.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MapCoderLLM=GPT-42024.05 | 93.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MapCoderBackbone=GPT-4-Turbo2024.12 | 93.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoderBackbone=GPT-4-Turbo2024.12 | 93.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HyEvoBackbone=gpt-4o-mini2026.03 | 93.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MIPROv2Source Model=GPT-4o, Target Model=o32025.12 | 93.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=true, Dec.=false2026.01 | 93.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=Qwen3-4B, Iterations=32025.11 | 93.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerBase model=GPT-OSS-120B, Ada.=true, DIM.=true, Dec.=false2026.01 | 93.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoder (CodeQwen1.5-7B-Chat)n=5, k=5, temperature=0.52024.12 | 93.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentDropoutBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=false, Dec.=true2026.01 | 93.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlowSteerFramework=Ours, Backbone=4o-mini2026.02 | 92.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentDropoutBase model=GPT-OSS-120B, Ada.=true, DIM.=false, Dec.=true2026.01 | 92.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaASLLM=gpt-4o-mini2025.09 | 92.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.43 | — | — | — | — | |
| MaASBackbone=gpt-4o-mini2026.03 | 92.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.12 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4oRepair strategy=+VER2026.04 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4on=164, Iterative Repair Setting=+VER2026.04 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5 OptimizerSource Model=GPT-4o, Target Model=o32025.12 | 92.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACES-CBackbone=Qwen2.5-Coder-14B2026.04 | 92.68 | — | — | 92.68 | 93.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACES-OBackbone=Qwen2.5-Coder-14B2026.04 | 92.68 | — | — | 93.29 | 93.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFA-MAS (pre-trained)Training protocol=One-for-All Topology Design (Pre-trained)2026.01 | 92.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4on=0, temperature=02024.12 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MURPHY - MARSBackbone=Qwen3-4B, Rollouts=72, Iterations=12025.11 | 92.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarmBase model=DeepSeek-V3.2-251201:671B, Ada.=true, DIM.=false, Dec.=false2026.01 | 92.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarmBase model=GPT-OSS-120B, Ada.=true, DIM.=false, Dec.=false2026.01 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct TransferSource Model=GPT-4o, Target Model=o32025.12 | 92.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoder (CodeQwen1.5-7B-Chat)n=2, k=5, temperature=0.52024.12 | 92.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o-2024-08-06Size Category=Closed-APIs2025.12 | 92.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.12 | 92.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude 3.5 Sonnet2024.07 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude 3.5 Sonnetshot=0-shot2024.07 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OneFlowModel=Claude 3.5 Haiku2026.01 | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoder (GPT-4-Turbo)n=5, k=5, temperature=0.52024.12 | 91.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAPSNumber of agents=52026.02 | 91.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Majority VotingBackbone=Qwen2.5-Coder-14B2026.04 | 91.46 | — | — | 92.68 | 93.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-DebateBase model=GPT-OSS-120B, Ada.=false, DIM.=false, Dec.=false2026.01 | 91.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SpecReasonBackbone=gpt-oss-120b2026.02 | 91.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4oModel Role=Source Model2025.12 | 91.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4oSource Model=GPT-4o, Evaluation Context=Source Performance Reference2025.12 | 91.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionLLM=GPT-42024.05 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionBackbone=GPT-42023.12 | 91 | — | 34.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlowNumber of agents=52026.02 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionBase Model=GPT-4, Evaluation Strategy=Agentic and Prompting Strategies2026.03 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |