Code Generation on CodeContests (test)
1,200Pass@1ALGO
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ALGOBase Model=ChatGPT, Synthesis Strategy=One-Time Synthesis, k=202023.05 | 1,200 | 1,200 | 14 | — | — | — | — | — | — | 7.6 | — | — | — | |
| ALGOBase Model=Codex, Synthesis Strategy=One-Time Synthesis, k=10002023.05 | 560 | 560 | 7.7 | — | 11.1 | — | — | — | — | 4.9 | — | — | — | |
| ChatGPTBase Model=ChatGPT, Synthesis Strategy=One-Time Synthesis, k=202023.05 | 440 | 662 | 12.21 | — | — | — | — | — | — | — | — | — | — | |
| CodeTBase Model=Codex, Synthesis Strategy=One-Time Synthesis, k=10002023.05 | 210 | 230 | 5.3 | — | 9.9 | — | — | — | — | 1.4 | — | — | — | |
| ALGOBase Model=GPT-2, Synthesis Strategy=Iterative Synthesis, k=202023.05 | 157 | 244 | 3.67 | — | 4.06 | — | — | — | — | 0.95 | — | — | — | |
| CodexBase Model=Codex, Synthesis Strategy=One-Time Synthesis, k=10002023.05 | 70 | 120 | 3 | — | 7.5 | — | — | — | — | — | — | — | — | |
| PG-TDBase Model=GPT-2, Synthesis Strategy=Iterative Synthesis, k=202023.05 | 67 | 112 | 2.53 | — | 3.81 | — | — | — | — | 0.05 | — | — | — | |
| PG-TD*Base Model=GPT-2, Synthesis Strategy=Iterative Synthesis, k=202023.05 | 62 | 107 | 2.26 | — | 3.45 | — | — | — | — | — | — | — | — | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=52024.12 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=22024.12 | 38.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentConductorSystem Category=Multi-Agent Systems with Topology Optimization, Model Scale=3B2026.02 | 38.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlowReasonerSystem Category=Multi-Agent Systems with Workflow Optimization2026.02 | 37.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MapCoderSystem Category=Classical Multi-Agent Systems2026.02 | 36.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaGPTSystem Category=Classical Multi-Agent Systems2026.02 | 35.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chain-of-AgentsSystem Category=Multi-Agent Systems with Workflow Optimization, Model Scale=32B2026.02 | 34.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoRe-CodeBased model=Q2.5-14B-CI2026.05 | 34.6 | — | — | — | — | — | — | — | — | — | 36.4 | 44.5 | — | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=12024.12 | 34.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRPOBased model=Q2.5-14B-CI2026.05 | 33.7 | — | — | — | — | — | — | — | — | — | 35.9 | 42.3 | — | |
| AgentPruneSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Complex2026.02 | 33.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CUREBased model=Q2.5-14B-CI2026.05 | 32.1 | — | — | — | — | — | — | — | — | — | 36.2 | 43.6 | — | |
| Focused-DPOBased model=Q2.5-14B-CI2026.05 | 31.7 | — | — | — | — | — | — | — | — | — | 34.4 | 47.2 | — | |
| GPTSwarmSystem Category=Multi-Agent Systems with Topology Optimization2026.02 | 31.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoRe-CodeBased model=Q3-4B2026.05 | 31.6 | — | — | — | — | — | — | — | — | — | 34.9 | 42.7 | — | |
| AgentPruneSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Layered2026.02 | 31.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeRL+Based model=Q2.5-14B-CI2026.05 | 29.2 | — | — | — | — | — | — | — | — | — | 33.5 | 47.7 | — | |
| Q2.5-14B-CIBased model=Q2.5-14B-CI2026.05 | 29.1 | — | — | — | — | — | — | — | — | — | 33.1 | 41.4 | — | |
| MacNetSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Layered2026.02 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MacNetSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Complex2026.02 | 28.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRPOBased model=Q3-4B2026.05 | 28.7 | — | — | — | — | — | — | — | — | — | 30.3 | 44.2 | — | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=52024.12 | 28.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeRL+Based model=Q2.5-7B-CI2026.05 | 28.5 | — | — | — | — | — | — | — | — | — | 30.3 | 40.7 | — | |
| CodeRL+Based model=Q3-4B2026.05 | 28.3 | — | — | — | — | — | — | — | — | — | 33.2 | 37.4 | — | |
| CoRe-CodeBased model=Q2.5-7B-CI2026.05 | 27.4 | — | — | — | — | — | — | — | — | — | 32.2 | 44.1 | — | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=22024.12 | 27.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerSystem Category=Multi-Agent Systems with Topology Optimization2026.02 | 26.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Q3-4BBased model=Q3-4B2026.05 | 26.3 | — | — | — | — | — | — | — | — | — | 31.5 | 27.7 | — | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=12024.12 | 26.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CUREBased model=Q2.5-7B-CI2026.05 | 25.9 | — | — | — | — | — | — | — | — | — | 29.4 | 39.2 | — | |
| Focused-DPOBased model=Q3-4B2026.05 | 25.4 | — | — | — | — | — | — | — | — | — | 30.3 | 29.8 | — | |
| GRPOBased model=Q2.5-7B-CI2026.05 | 24.5 | — | — | — | — | — | — | — | — | — | 28.5 | 43.3 | — | |
| CUREBased model=Q3-4B2026.05 | 24.2 | — | — | — | — | — | — | — | — | — | 28.7 | 33.9 | — | |
| Focused-DPOBased model=Q2.5-7B-CI2026.05 | 22.7 | — | — | — | — | — | — | — | — | — | 27.2 | 36.7 | — | |
| AFlowSystem Category=Multi-Agent Systems with Workflow Optimization2026.02 | 21.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Q2.5-7B-CIBased model=Q2.5-7B-CI2026.05 | 21.3 | — | — | — | — | — | — | — | — | — | 24.4 | 33.9 | — | |
| DSCV2-16BBased model=DSCV2-16B2026.05 | 21.3 | — | — | — | — | — | — | — | — | — | 28.5 | 34.4 | — | |
| ThinkCoderBackbone=GPT-4o, Self-refinement budget (n)=02024.12 | 21 | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoGenSystem Category=Classical Multi-Agent Systems2026.02 | 20.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Q2.5-7B-IBased model=Q2.5-7B-I2026.05 | 19.4 | — | — | — | — | — | — | — | — | — | 21.9 | 27.2 | — | |
| GPT-4o-miniSystem Category=Vanilla, Backbone=GPT-4o-mini2026.02 | 18.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Sequence-level adapterSampling strategy=w/o budget, Setting=mix CoT, Training domain=MATH2026.03 | 17.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ThinkCoderBackbone=GPT-4-Turbo, Self-refinement budget (n)=02024.12 | 16.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Static samplingSampling strategy=Mixed, Setting=mix CoT2026.03 | 14.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Static samplingSampling strategy=Best, Setting=mix CoT2026.03 | 13.97 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Sequence-level adapterSampling strategy=w/o budget, Setting=w/o CoT, Training domain=MATH2026.03 | 12.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MPSC-LabelFoundation Model=GPT-3.5-Turbo2023.09 | 11.94 | 15.55 | — | — | — | — | — | — | — | — | 18.2 | — | — | |
| Static samplingSampling strategy=Best, Setting=w/o CoT2026.03 | 11.43 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Static samplingSampling strategy=Mixed, Setting=w/o CoT2026.03 | 10.53 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MPSC-SemanticFoundation Model=GPT-3.5-Turbo2023.09 | 10.09 | 10.29 | — | — | — | — | — | — | — | — | 10.3 | — | — | |
| CodeTFoundation Model=GPT-3.5-Turbo2023.09 | 9.92 | 10.18 | — | — | — | — | — | — | — | — | 10.3 | — | — | |
| MBR-EXECFoundation Model=GPT-3.5-Turbo2023.09 | 8.25 | 8.87 | — | — | — | — | — | — | — | — | 9.08 | — | — | |
| Self-consistencyFoundation Model=GPT-3.5-Turbo2023.09 | 8.1 | 8.42 | — | — | — | — | — | — | — | — | 8.48 | — | — | |
| GPT-42023.09 | 6.1 | 8.28 | — | — | — | — | — | — | — | — | 11.72 | — | — | |
| MPSC-LexicalFoundation Model=GPT-3.5-Turbo2023.09 | 5.45 | 5.45 | — | — | — | — | — | — | — | — | 6.06 | — | — | |
| MPSC-UniformFoundation Model=GPT-3.5-Turbo2023.09 | 4.71 | 6.65 | — | — | — | — | — | — | — | — | 8.31 | — | — | |
| GPT-3.5-Turbo2023.09 | 2.57 | 4.22 | — | — | — | — | — | — | — | — | 7.16 | — | — | |
| WizardCoder2023.09 | 2.15 | 3.4 | — | — | — | — | — | — | — | — | 5.37 | — | — | |
| CODETModel=code-davinci-002, Setting=zero-shot, Temperature=0.8, Sampling number=10002022.07 | 2.1 | 2.3 | 5.3 | — | 9.9 | — | — | — | — | — | — | — | — | |
| BaselineModel=code-davinci-002, Setting=zero-shot, Temperature=0.8, Sampling number=10002022.07 | 0.7 | — | 3 | 5.7 | 7.5 | 13.9 | — | — | — | — | — | — | — | |
| AlphaCodeModel size=9B, Clustering=false2022.02 | — | — | — | — | — | — | 14.3 | 21.5 | 25.8 | — | — | — | — | |
| AlphaCodeModel size=41B, Clustering=false2022.02 | — | — | — | — | — | — | 15.6 | 23.2 | 27.7 | — | — | — | — | |
| AlphaCodeModel size=41B, Clustering=true2022.02 | — | — | — | — | — | — | 16.4 | 25.4 | 29.6 | — | — | — | — | |
| CPPOBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 24.3 | |
| CPPOBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 43.6 | |
| Direct SolveBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 9.4 | |
| Direct SolveBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 17.5 | |
| Pass@K Training / RLVRBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 11.5 | |
| Pass@K Training / RLVRBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 18.5 | |
| PKPOBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 15.6 | |
| PKPOBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 18.3 | |
| Plan-and-SolveBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 9.8 | |
| Plan-and-SolveBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 17.1 | |
| PlanSearchBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 12.8 | |
| PlanSearchBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 16.8 | |
| Tuple Planner SFTBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 14.2 | |
| Tuple Planner SFTBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 28.5 | |
| UpSkillBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 12.1 | |
| UpSkillBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | 13.6 |