Code Generation on HumanEval (test)
100Pass@1Claude 3.5 Sonnet
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude 3.5 SonnetPrompting Strategy=Hierarchical Prompting Framework (HPF)2024.06 | 100 | — | — | — | 1.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-3 8BPrompting Strategy=Hierarchical Prompting Framework (HPF)2024.06 | 100 | — | — | — | 1.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentConductorSystem Category=Multi-Agent Systems with Topology Optimization, Model Scale=3B2026.02 | 97.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlowReasonerSystem Category=Multi-Agent Systems with Workflow Optimization2026.02 | 97.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentPruneSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Layered2026.02 | 96.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MapCoderSystem Category=Classical Multi-Agent Systems2026.02 | 96.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentPruneSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Complex2026.02 | 96.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral-Nemo 12BPrompting Strategy=Hierarchical Prompting Framework (HPF)2024.06 | 96 | — | — | — | 2.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaGPTSystem Category=Classical Multi-Agent Systems2026.02 | 95.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MacNetSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Complex2026.02 | 95.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerSystem Category=Multi-Agent Systems with Topology Optimization2026.02 | 95.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MANGOBase LLM=GPT-4o-mini2026.05 | 95.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chain-of-AgentsSystem Category=Multi-Agent Systems with Workflow Optimization, Model Scale=32B2026.02 | 95.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MacNetSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Layered2026.02 | 95.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4oPrompting Strategy=Hierarchical Prompting Framework (HPF)2024.06 | 95 | — | — | — | 2.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarmSystem Category=Multi-Agent Systems with Topology Optimization2026.02 | 94.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MANGO (Skip-2)Base LLM=GPT-4o-mini2026.05 | 94.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FUNCODERModel=GPT-42024.05 | 94.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | 11.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlowSystem Category=Multi-Agent Systems with Workflow Optimization2026.02 | 94.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral 7BPrompting Strategy=Hierarchical Prompting Framework (HPF)2024.06 | 93 | — | — | — | 1.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaASBase LLM=GPT-4o-mini2026.05 | 92.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LATS (ReAct)Model=GPT-42023.10 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemma-2 9BPrompting Strategy=Hierarchical Prompting Framework (HPF)2024.06 | 91 | — | — | — | 1.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionModel=GPT-4, Strategy=Reflexion, Zero-shot=true2023.03 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionLLM=GPT-42023.05 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionModel=GPT-42023.10 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReflexionModel=GPT-42024.05 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | 8.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlowBase LLM=GPT-4o-mini2026.05 | 90.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeTModel=GPT-42024.05 | 90.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | 8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.42025.05 | 90.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DyLANBase LLM=GPT-4o-mini2026.05 | 90.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoGenSystem Category=Classical Multi-Agent Systems2026.02 | 90.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.52025.05 | 89.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.72025.05 | 89.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarmBase LLM=GPT-4o-mini2026.05 | 89.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.12025.05 | 89.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.4, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 89.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Structural, Variant=Str-32025.05 | 89.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentSquareBase LLM=GPT-4o-mini2026.05 | 89.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.32025.05 | 88.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.3, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 88.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.22025.05 | 88.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.5, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 88.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Structural, Variant=Str-22025.05 | 88.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Plan-and-ExecuteBase LLM=GPT-4o-mini2026.05 | 88.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Structural, Variant=Str-12025.05 | 88.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM-DebateBase LLM=GPT-4o-mini2026.05 | 88.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-1.62025.05 | 88.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SC (CoT×5)Base LLM=GPT-4o-mini2026.05 | 88.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACBackbone=GPT-4o-mini, Optimization Dimension=Functional, Variant=Fun-22025.05 | 88.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.1, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 88.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.6, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 88.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.7, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 88.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultiPersonaBase LLM=GPT-4o-mini2026.05 | 88.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoTBase LLM=GPT-4o-mini2026.05 | 88.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Self-RefineBase LLM=GPT-4o-mini2026.05 | 87.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o-miniSystem Category=Vanilla, Backbone=GPT-4o-mini2026.02 | 87.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DyLANBackbone=GPT-4o-mini, Category=Baseline2025.05 | 87.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Structural Dimension, Optimization Sub-dimension=Str-3, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 87.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlowBackbone=GPT-4o-mini, Category=Baseline2025.05 | 87.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IO (direct LLM invocation)Base LLM=GPT-4o-mini2026.05 | 87.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Structural Dimension, Optimization Sub-dimension=Str-2, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 86.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-2, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 86.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Structural Dimension, Optimization Sub-dimension=Str-1, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 86.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ANPLLLM=GPT-42023.05 | 86.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OMACOptimization Dimension=Functional Dimension, Optimization Sub-dimension=Fun-1.2, Base LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 86.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaGPTModel=GPT-42024.05 | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DyLANBase LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 85.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlowBase LLM=gpt-3.5-turbo-1106, Temperature=0.82025.05 | 85.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4Decoding Strategy=greedy2023.12 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FUNCODERModel=GPT-3.52024.05 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACES-C + DS3Post-hoc reranking type=additional information, n=≈200, m=≈500, reranked_model=GPT-3.5-Turbo2026.04 | 85.37 | — | — | — | — | 85.98 | — | 87.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Parsel + CodeTLLM=GPT-42023.05 | 85.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ParselModel=GPT-42024.05 | 85 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentVerseBackbone=GPT-4o-mini, Category=Baseline2025.05 | 84.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MPSC-LabelFoundation Model=GPT-3.5-Turbo2023.09 | 84.29 | — | — | — | — | 86.79 | — | 87.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADASBase LLM=GPT-4o-mini2026.05 | 84.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACES-OPost-hoc reranking type=only code x test execution, n=≈200, m=≈500, reranked_model=GPT-3.5-Turbo2026.04 | 84.15 | — | — | — | — | 85.98 | — | 86.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADASBackbone=GPT-4o-mini, Category=Baseline2025.05 | 83.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LATS (ReAct)Model=GPT-3.52023.10 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACES-O + DS3Post-hoc reranking type=additional information, n=≈200, m=≈500, reranked_model=GPT-3.5-Turbo2026.04 | 83.54 | — | — | — | — | 83.54 | — | 85.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TACBackbone=Qwen3-4B-Base, Seeds=12026.06 | 83.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MPSC-SemanticFoundation Model=GPT-3.5-Turbo2023.09 | 83.38 | — | — | — | — | 84.25 | — | 84.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RandomBackbone=Qwen3-4B-Base, Seeds=12026.06 | 83.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UTS-guided systemT=0.5, N=1602026.02 | 82.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23 | 1.643 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACES-CPost-hoc reranking type=only code x test execution, n=≈200, m=≈500, reranked_model=GPT-3.5-Turbo2026.04 | 82.93 | — | — | — | — | 82.93 | — | 82.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LDBModel=GPT-3.52024.05 | 82.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | 14.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StandardModel=GPT-42024.05 | 82.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MPSC-LexicalFoundation Model=GPT-3.5-Turbo2023.09 | 82.32 | — | — | — | — | 84.76 | — | 86.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Sub-block-basedModel=NBDiff, Cache=None2026.01 | 82.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Original [7]LLM=GPT-42023.05 | 82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DS3Post-hoc reranking type=additional information, n=≈200, m=≈500, reranked_model=GPT-3.5-Turbo2026.04 | 81.71 | — | — | — | — | 82.32 | — | 82.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DCDModel=NBDiff, Cache=None, Time=5261632026.01 | 81.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-42023.09 | 81.48 | — | — | — | — | 86.31 | — | 90.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4Direct inference=true2026.04 | 81.48 | — | — | — | — | 86.31 | — | 90.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeTModel=GPT-3.52024.05 | 81.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | 12.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UTS-guided systemT=1.0, N=1602026.02 | 81.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 42 | 1.636 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SECBackbone=Qwen3-4B-Base, Seeds=12026.06 | 81.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FUNCODERModel=StableCode-3b2024.05 | 81 | — | — | — | — | — | — | — | — | — | — | — | — | — | 20 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Majority VotingPost-hoc reranking type=only code x test execution, n=≈200, m=≈500, reranked_model=GPT-3.5-Turbo2026.04 | 80.49 | — | — | — | — | 82.93 | — | 83.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |