Code Generation on HumanEval (Pass@k, AUC, and Token Usage)
94.1Pass@1ExPairT-LLM
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ExPairT-LLMModel=CodeGen2025.11 | 94.1 | — | — | — | — | — | |
| Vanilla (Ref)LLM=Claude-3.5-Sonnet, Multi-agent=No, Dynamic Routing=No2026.03 | 93.7 | — | — | — | — | — | |
| MaASDyn.=true, Dis.=false2025.11 | 92.85 | — | — | — | — | — | |
| AMRO-SLLM=LLM Pool*, Multi-agent=Yes, Dynamic Routing=Yes2026.03 | 92.2 | — | — | — | — | — | |
| OneFlowMethod category=Single-LLM implementation, Executor LLM=GPT-4o-mini, Execution protocol=single-agent execution2026.01 | 92.1 | — | — | — | — | — | |
| ExPairT-LLMModel=StarCoder2025.11 | 91.7 | — | — | — | — | — | |
| OneFlowMethod category=Automatically designed multi-agent frameworks, Executor LLM=GPT-4o-mini, Execution protocol=multi-agent2026.01 | 91.6 | — | — | — | — | — | |
| ExPairT-LLMModel=DeepSeek-Coder2025.11 | 91.6 | — | — | — | — | — | |
| BiRouterDyn.=true, Dis.=true2025.11 | 91.46 | — | — | — | — | — | |
| ExPairT-LLMModel=Codex2025.11 | 91.3 | — | — | — | — | — | |
| AFlowMethod category=Single-LLM implementation, Executor LLM=GPT-4o-mini, Execution protocol=single-agent execution2026.01 | 91.1 | — | — | — | — | — | |
| MasRouterLLM=LLM Pool*, Multi-agent=Yes, Dynamic Routing=Yes2026.03 | 90.62 | — | — | — | — | — | |
| DyLANDyn.=true, Dis.=false2025.11 | 90.42 | — | — | — | — | — | |
| CoTMethod category=Manual baselines, Executor LLM=GPT-4o-mini, Execution protocol=standard2026.01 | 90.3 | — | — | — | — | — | |
| Vanilla (Ref)LLM=GPT-4o, Multi-agent=No, Dynamic Routing=No2026.03 | 90.2 | — | — | — | — | — | |
| AFlowMethod category=Automatically designed multi-agent frameworks, Executor LLM=GPT-4o-mini, Execution protocol=multi-agent2026.01 | 90.1 | — | — | — | — | — | |
| AFlowLLM=GPT-4o-mini, Multi-agent=Yes, Dynamic Routing=No2026.03 | 90.06 | — | — | — | — | — | |
| CoT SCMethod category=Manual baselines, Executor LLM=GPT-4o-mini, Execution protocol=standard, shots=5-shot2026.01 | 89.8 | — | — | — | — | — | |
| GPTSwarmDyn.=false, Dis.=false2025.11 | 89.32 | — | — | — | — | — | |
| AgentVerseDyn.=true, Dis.=false2025.11 | 89.29 | — | — | — | — | — | |
| IOMethod category=Manual baselines, Executor LLM=GPT-4o-mini, Execution protocol=standard2026.01 | 89.1 | — | — | — | — | — | |
| MultiPersonaMethod category=Manual baselines, Executor LLM=GPT-4o-mini, Execution protocol=standard2026.01 | 89.1 | — | — | — | — | — | |
| ExPairT-LLMModel=CodeLlama2025.11 | 89.1 | — | — | — | — | — | |
| LLM-BlenderDyn.=false, Dis.=false2025.11 | 88.8 | — | — | — | — | — | |
| LLM-DebateDyn.=false, Dis.=false2025.11 | 88.68 | — | — | — | — | — | |
| SC (CoTx5)2025.11 | 88.6 | — | — | — | — | — | |
| CoT2025.11 | 88.13 | — | — | — | — | — | |
| BaseBackbone=Qwen3-8B2025.09 | 88 | — | — | — | — | — | |
| GRPOBackbone=Qwen3-8B2025.09 | 87.8 | — | — | — | — | — | |
| RouterDCLLM=LLM Pool*, Multi-agent=No, Dynamic Routing=Yes2026.03 | 87.75 | — | — | — | — | — | |
| G-DesignerDyn.=false, Dis.=false2025.11 | 87.5 | — | — | — | — | — | |
| AoT (Atom)LLM=GPT-4o-mini, Multi-agent=No, Dynamic Routing=No2026.03 | 87.5 | — | — | — | — | — | |
| ComplexCoT2025.11 | 87.49 | — | — | — | — | — | |
| Single-agent2025.11 | 87.08 | — | — | — | — | — | |
| IPOBackbone=Qwen3-8B2025.09 | 86.8 | — | — | — | — | — | |
| CoTLLM=GPT-4o-mini, Multi-agent=No, Dynamic Routing=No2026.03 | 86.69 | — | — | — | — | — | |
| VanillaLLM=Claude-3.5-haiku, Multi-agent=No, Dynamic Routing=No2026.03 | 86.4 | — | — | — | — | — | |
| GPTSwarmLLM=GPT-4o-mini, Multi-agent=Yes, Dynamic Routing=No2026.03 | 86.28 | — | — | — | — | — | |
| GoT (Graph)LLM=GPT-4o-mini, Multi-agent=No, Dynamic Routing=No2026.03 | 85.8 | — | — | — | — | — | |
| VanillaLLM=GPT-4o-mini, Multi-agent=No, Dynamic Routing=No2026.03 | 85.71 | — | — | — | — | — | |
| AFlowLLM=Gemini-1.5-flash, Multi-agent=Yes, Dynamic Routing=No2026.03 | 85.69 | — | — | — | — | — | |
| ToT (Tree)LLM=GPT-4o-mini, Multi-agent=No, Dynamic Routing=No2026.03 | 85.2 | — | — | — | — | — | |
| MacNetDyn.=false, Dis.=false2025.11 | 84.57 | — | — | — | — | — | |
| LLM-DebateLLM=GPT-4o-mini, Multi-agent=Yes, Dynamic Routing=No2026.03 | 84.38 | — | — | — | — | — | |
| RouteLLMLLM=LLM Pool*, Multi-agent=No, Dynamic Routing=Yes2026.03 | 83.85 | — | — | — | — | — | |
| VanillaLLM=Gemini-1.5-flash, Multi-agent=No, Dynamic Routing=No2026.03 | 82.61 | — | — | — | — | — | |
| IPOBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 82.6 | — | — | — | — | — | |
| SafeKeyBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 81.9 | — | — | — | — | — | |
| BaseBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 81.7 | — | — | — | — | — | |
| SafeKeyBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 81 | — | — | — | — | — | |
| GRPOBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 81 | — | — | — | — | — | |
| CODETModel=DeepSeek-Coder2025.11 | 80.9 | — | — | — | — | — | |
| B^4Model=DeepSeek-Coder2025.11 | 80.8 | — | — | — | — | — | |
| B^4Model=Codex2025.11 | 80.5 | — | — | — | — | — | |
| VanillaLLM=Llama-3.1-70b, Multi-agent=No, Dynamic Routing=No2026.03 | 80.5 | — | — | — | — | — | |
| STARBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 80 | — | — | — | — | — | |
| GRPOBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 79.9 | — | — | — | — | — | |
| BaseBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 79.5 | — | — | — | — | — | |
| IPOBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 79.5 | — | — | — | — | — | |
| RealSafeBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 79.2 | — | — | — | — | — | |
| RealSafeBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 77.8 | — | — | — | — | — | |
| SafeChainBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 77.4 | — | — | — | — | — | |
| STARBackbone=DeepSeek-R1-Distill-Llama-8B2025.09 | 77.1 | — | — | — | — | — | |
| ReasoningBackbone=Qwen2.5-7B family2026.01 | 76.63 | — | — | — | — | — | |
| SafeChainBackbone=DeepSeek-R1-Distill-Qwen-7B2025.09 | 76.5 | — | — | — | — | — | |
| CODETModel=Codex2025.11 | 74.1 | — | — | — | — | — | |
| Claude-Haiku-4.5Category=Candidate Models2026.03 | 73.7 | — | — | — | — | — | |
| B^4Model=StarCoder2025.11 | 70.5 | — | — | — | — | — | |
| FineRouterCategory=Routers2026.03 | 68.4 | — | — | — | — | — | |
| TI-DPOModel=LLaMA-3.2-3B2025.05 | 68 | — | — | — | — | — | |
| B^4Model=CodeLlama2025.11 | 66.6 | — | — | — | — | — | |
| CODETModel=StarCoder2025.11 | 65.4 | — | — | — | — | — | |
| OriginalModel=DeepSeek-Coder2025.11 | 65.4 | — | — | — | — | — | |
| B^4Model=CodeGen2025.11 | 64 | — | — | — | — | — | |
| TPOModel=LLaMA-3.2-3B2025.05 | 64 | — | — | — | — | — | |
| GRPOModel=LLaMA-3.2-3B2025.05 | 63.8 | — | — | — | — | — | |
| CPOModel=LLaMA-3.2-3B2025.05 | 63.5 | — | — | — | — | — | |
| Claude-Sonnet-4.5Category=Candidate Models2026.03 | 63.2 | — | — | — | — | — | |
| MLPCategory=Routers2026.03 | 63.2 | — | — | — | — | — | |
| KTOModel=LLaMA-3.2-3B2025.05 | 63 | — | — | — | — | — | |
| CODETModel=CodeLlama2025.11 | 62.5 | — | — | — | — | — | |
| DPOModel=LLaMA-3.2-3B2025.05 | 62 | — | — | — | — | — | |
| TDPOModel=LLaMA-3.2-3B2025.05 | 62 | — | — | — | — | — | |
| ReasonAnyBackbone=Qwen2.5-7B family2026.01 | 61.71 | — | — | — | — | — | |
| SIMPOModel=LLaMA-3.2-3B2025.05 | 61.5 | — | — | — | — | — | |
| SFTModel=LLaMA-3.2-3B2025.05 | 61 | — | — | — | — | — | |
| IPOModel=LLaMA-3.2-3B2025.05 | 59 | — | — | — | — | — | |
| Mistral-LargeCategory=Candidate Models2026.03 | 57.9 | — | — | — | — | — | |
| Dele-SimKOBackbone=Qwen3-8B2026.01 | 57.1 | 80.5 | 83.5 | 73.2 | 5.8 | — | |
| Dele-GRPOBackbone=Qwen3-8B2026.01 | 56.4 | 79.8 | 82.5 | 71.2 | 5.9 | — | |
| LCoT-SimKOBackbone=Qwen3-8B2026.01 | 56.2 | 79.2 | 82 | 71.8 | 6 | — | |
| TGR-LatentBackbone=Qwen3-8B2026.01 | 56.1 | 81.1 | 84.2 | 73.5 | 4.6 | — | |
| TGR-TokenBackbone=Qwen3-8B2026.01 | 55.3 | 78.5 | 81.8 | 70.8 | 4.4 | — | |
| CODETModel=CodeGen2025.11 | 55.1 | — | — | — | — | — | |
| LCoT-GRPOBackbone=Qwen3-8B2026.01 | 55 | 76.8 | 79.5 | 69.5 | 6 | — | |
| PSamplingBackbone=Qwen3-8B2026.01 | 52.8 | 74.5 | 77.8 | 67.2 | 3.6 | — | |
| Qwen3-235B-A22BCategory=Candidate Models2026.03 | 52.6 | — | — | — | — | — | |
| DAREBackbone=Qwen2.5-7B family2026.01 | 49.4 | — | — | — | — | — | |
| IOA (Ours)Model=Qwen2.5-7B2026.02 | 47.67 | — | — | — | — | — | |
| DeepSeek-R1Category=Candidate Models2026.03 | 47.4 | — | — | — | — | — |