Code Generation on APPS (test)
56.3Introductory ScoreCode Llama
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Code LlamaModel Size=34B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 56.3 | — | — | — | — | — | — | — | 2,430 | 1,540 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=34B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 55.7 | — | — | — | — | — | — | — | 2,280 | 1,640 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=34B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 54.9 | — | — | — | — | — | — | — | 2,390 | 1,680 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=13B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 51.6 | — | — | — | — | — | — | — | 2,150 | 1,460 | — | — | — | — | — | — | |
| Code LlamaModel Size=13B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 49 | — | — | — | — | — | — | — | 1,840 | 1,200 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=13B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 48.7 | — | — | — | — | — | — | — | 1,960 | 1,310 | — | — | — | — | — | — | |
| Code LlamaModel Size=34B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 39 | — | — | — | — | — | — | — | 1,220 | 470 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=7B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 38.3 | — | — | — | — | — | — | — | 1,490 | 910 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=34B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 37.8 | — | — | — | — | — | — | — | 1,110 | 510 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=34B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 35.9 | — | — | — | — | — | — | — | 1,110 | 550 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=7B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 35.4 | — | — | — | — | — | — | — | 940 | 850 | — | — | — | — | — | — | |
| Code LlamaModel Size=7B, Pass@=100, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 33.5 | — | — | — | — | — | — | — | 940 | 710 | — | — | — | — | — | — | |
| Code LlamaModel Size=34B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 32.8 | — | — | — | — | — | — | — | 880 | 290 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=13B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 32.8 | — | — | — | — | — | — | — | 1,000 | 430 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=34B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 31.6 | — | — | — | — | — | — | — | 790 | 320 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=13B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 30.3 | — | — | — | — | — | — | — | 960 | 380 | — | — | — | — | — | — | |
| Code LlamaModel Size=13B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 30.2 | — | — | — | — | — | — | — | 810 | 340 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=34B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 28.9 | — | — | — | — | — | — | — | 780 | 350 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=13B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 26.3 | — | — | — | — | — | — | — | 710 | 280 | — | — | — | — | — | — | |
| CodexModel Size=12B, Pass@=1000, Protocol=one-shot2023.08 | 25 | — | — | — | — | — | — | — | 370 | 320 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=13B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 24 | — | — | — | — | — | — | — | 690 | 240 | — | — | — | — | — | — | |
| Code LlamaModel Size=13B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 23.7 | — | — | — | — | — | — | — | 560 | 210 | — | — | — | — | — | — | |
| AlphaCode (Filtered 50000)Model Size=1B, Pass@=5, Protocol=fine-tuned2023.08 | 20.4 | — | — | — | — | — | — | — | 970 | 780 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=7B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 18.5 | — | — | — | — | — | — | — | 630 | 220 | — | — | — | — | — | — | |
| AlphaCode (Filtered 10000)Model Size=1B, Pass@=5, Protocol=fine-tuned2023.08 | 18.2 | — | — | — | — | — | — | — | 820 | 670 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=7B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 17.9 | — | — | — | — | — | — | — | 310 | 200 | — | — | — | — | — | — | |
| AlphaCodeModel Size=1B, Pass@=1000, Protocol=fine-tuned2023.08 | 17.7 | — | — | — | — | — | — | — | 520 | 710 | — | — | — | — | — | — | |
| Code LlamaModel Size=7B, Pass@=10, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 15.6 | — | — | — | — | — | — | — | 310 | 140 | — | — | — | — | — | — | |
| AlphaCode (Filtered 1000)Model Size=1B, Pass@=5, Protocol=fine-tuned2023.08 | 14.4 | — | — | — | — | — | — | — | 560 | 460 | — | — | — | — | — | — | |
| Code Llama - InstructModel Size=7B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 12.9 | — | — | — | — | — | — | — | 210 | 110 | — | — | — | — | — | — | |
| Code Llama - PythonModel Size=7B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 12.7 | — | — | — | — | — | — | — | 420 | 130 | — | — | — | — | — | — | |
| Code LlamaModel Size=7B, Pass@=5, Protocol=two-shot, Sampling=nucleus p=0.95, temp=0.62023.08 | 10.8 | — | — | — | — | — | — | — | 200 | 80 | — | — | — | — | — | — | |
| CodexModel Size=12B, Pass@=5, Protocol=one-shot2023.08 | 9.7 | — | — | — | — | — | — | — | 50 | 10 | — | — | — | — | — | — | |
| GPT-NeoModel Size=2.7B, Pass@=5, Protocol=fine-tuned2023.08 | 5.5 | — | — | — | — | — | — | — | 80 | 0 | — | — | — | — | — | — | |
| CodexModel Size=12B, Pass@=1, Protocol=one-shot2023.08 | 4.1 | — | — | — | — | — | — | — | 10 | 0 | — | — | — | — | — | — | |
| GPT-NeoModel Size=2.7B, Pass@=1, Protocol=fine-tuned2023.08 | 3.9 | — | — | — | — | — | — | — | 60 | 0 | — | — | — | — | — | — | |
| AFlowSystem Category=Multi-Agent Systems with Workflow Optimization2026.02 | — | — | 35.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFlowPrompt Tokens=531450, Completion Tokens=184800, S_complex=3.72026.02 | — | — | 35.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentConductorSystem Category=Multi-Agent Systems with Topology Optimization, Model Scale=3B2026.02 | — | — | 58.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentConductorModel Size=3B, Prompt Tokens=277600, Completion Tokens=79800, S_complex=5.22026.02 | — | — | 58.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentPruneSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Complex2026.02 | — | — | 38.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentPruneSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Layered2026.02 | — | — | 39.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AgentPruneTopology=Layered, Prompt Tokens=364950, Completion Tokens=141150, S_complex=3.82026.02 | — | — | 39.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoGenSystem Category=Classical Multi-Agent Systems2026.02 | — | — | 23.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base modelSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 10.51 | |
| Chain-of-AgentsSystem Category=Multi-Agent Systems with Workflow Optimization, Model Scale=32B2026.02 | — | — | 41.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chain-of-AgentsModel Size=32B, Prompt Tokens=334650, Completion Tokens=134250, S_complex=4.12026.02 | — | — | 41.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CODERANKERCode gen. model=Codex2022.06 | — | — | — | 4.5 | — | 10.2 | — | 73.4 | — | — | — | — | — | — | — | — | |
| CODERANKERCode gen. model=GPT-J2022.06 | — | — | — | 0.8 | — | 2.6 | — | 63.8 | — | — | — | — | — | — | — | — | |
| CODERANKERCode gen. model=GPT-Neo 1.3B2022.06 | — | — | — | 0.3 | — | 1.1 | — | 73.7 | — | — | — | — | — | — | — | — | |
| CODERANKERCode gen. model=GPT-Neo 125M2022.06 | — | — | — | 0.1 | — | 0.5 | — | 43.9 | — | — | — | — | — | — | — | — | |
| CodeTModel=Codex002, Evaluation Protocol=zero-shot2023.10 | — | — | — | — | — | — | — | — | — | — | 34.6 | 8.1 | 220 | — | — | — | |
| CodexCode gen. model=Codex2022.06 | — | 24.1 | 3.8 | — | 9.2 | — | 59.6 | — | — | — | — | — | — | — | — | — | |
| CPPOBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 79.4 | — | |
| CPPOBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.6 | — | |
| DEARSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16.9 | |
| Dense baselineModel Type=Dense, Compute=5.45e212025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.35 | — | — | |
| Direct SolveBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 51.5 | — | |
| Direct SolveBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 69.6 | — | |
| FlowReasonerSystem Category=Multi-Agent Systems with Workflow Optimization2026.02 | — | — | 39.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlowReasonerPrompt Tokens=437250, Completion Tokens=148050, S_complex=2.42026.02 | — | — | 39.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerSystem Category=Multi-Agent Systems with Topology Optimization2026.02 | — | — | 37.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-DesignerPrompt Tokens=320550, Completion Tokens=139200, S_complex=3.62026.02 | — | — | 37.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o-miniSystem Category=Vanilla, Backbone=GPT-4o-mini2026.02 | — | — | 20.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-JCode gen. model=GPT-J2022.06 | — | 7.2 | 0.5 | — | 1.6 | — | 45.4 | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 1.3BCode gen. model=GPT-Neo 1.3B2022.06 | — | 3 | 0.14 | — | 0.53 | — | 35.2 | — | — | — | — | — | — | — | — | — | |
| GPT-Neo 125MCode gen. model=GPT-Neo 125M2022.06 | — | 1.5 | 0.04 | — | 0.17 | — | 28.5 | — | — | — | — | — | — | — | — | — | |
| GPTSwarmSystem Category=Multi-Agent Systems with Topology Optimization2026.02 | — | — | 36.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTSwarmPrompt Tokens=381450, Completion Tokens=155400, S_complex=3.52026.02 | — | — | 36.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GraphDPOground-truth anchoring=false2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 72.93 | — | — | |
| GraphDPOground-truth anchoring=true2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 73.76 | — | — | |
| GreedyModel=Codex002, Evaluation Protocol=zero-shot2023.10 | — | — | — | — | — | — | — | — | — | — | 27.2 | 5.1 | 180 | — | — | — | |
| GRPO2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.83 | — | — | |
| LiPO2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 69.32 | — | — | |
| MacNetSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Complex2026.02 | — | — | 37.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MacNetSystem Category=Multi-Agent Systems with Topology Optimization, Topology Variant=Layered2026.02 | — | — | 36.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MacNetTopology=Layered, Prompt Tokens=472950, Completion Tokens=200100, S_complex=2.92026.02 | — | — | 36.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MapCoderSystem Category=Classical Multi-Agent Systems2026.02 | — | — | 40.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaGPTSystem Category=Classical Multi-Agent Systems2026.02 | — | — | 51.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MoE w/ optimal ARModel Type=MoE, Activation rate=20.07, Compute=2.86e21, Data reuse=-2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 6.8 | — | — | |
| MoE w/ optimal ARModel Type=MoE, Activation rate=20.07, Compute=2.86e21, Data reuse=strict2025.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | 8.18 | — | — | |
| MPO2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.23 | — | — | |
| Offline KDSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 5.03 | |
| OTS2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 44.22 | — | — | |
| Pass@K Training / RLVRBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60.7 | — | |
| Pass@K Training / RLVRBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.2 | — | |
| PKPOBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 72.2 | — | |
| PKPOBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.7 | — | |
| Plan-and-SolveBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 53 | — | |
| Plan-and-SolveBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.1 | — | |
| PlanSearchBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 55.4 | — | |
| PlanSearchBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 77 | — | |
| PRO2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.58 | — | — | |
| RandomModel=Codex002, Evaluation Protocol=zero-shot2023.10 | — | — | — | — | — | — | — | — | — | — | 20.35 | 3.11 | 74 | — | — | — | |
| SFT2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.08 | — | — | |
| SPIN2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.87 | — | — | |
| SRankModel=Codex002, Evaluation Protocol=zero-shot2023.10 | — | — | — | — | — | — | — | — | — | — | 37.79 | 9.53 | 329 | — | — | — | |
| Standard OPDSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 14.34 | |
| SWEPO2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.36 | — | — | |
| Tuple Planner SFTBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 52.8 | — |