Code Generation on APPS Introductory
92.1pass@1LCP
Evaluation Results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LCPLLM backbone=GPT-4o2026.03 | 92.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LPWLLM backbone=GPT-4o2026.03 | 87.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LDBLLM backbone=GPT-4o2026.03 | 78.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineLLM backbone=GPT-4o2026.03 | 63.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LogitsCoderModel Category=Decoding-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | 62.21 | — | |
| LATSModel Category=Search-guided Reasoning Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 35 | — | — | — | — | — | — | — | — | — | — | — | — | 53.21 | — | |
| Contrastive DecodingModel Category=Decoding-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 35 | — | — | — | — | — | — | — | — | — | — | — | — | 56.23 | — | |
| CODETModel=code-davinci-002, Setting=zero-shot, Temperature=0.8, Sampling number=502022.07 | 34.6 | — | — | — | — | — | — | — | 41.2 | 53.2 | — | — | — | — | — | |
| MCTSModel Category=Search-guided Reasoning Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | 56.27 | — | |
| Guided DecodingModel Category=Decoding-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | 56.15 | — | |
| self-playModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 31 | — | — | — | — | — | — | — | — | — | — | — | — | 51.46 | — | |
| RethinkMCTSModel Category=Search-guided Reasoning Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 31 | — | — | — | — | — | — | — | — | — | — | — | — | 59.95 | — | |
| BaselineModel=code-davinci-002, Setting=zero-shot, Temperature=0.8, Sampling number=502022.07 | 27.2 | — | — | — | — | — | — | — | — | 46.6 | 59.4 | — | — | — | — | |
| ReflexionModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 27 | — | — | — | — | — | — | — | — | — | — | — | — | 45.38 | — | |
| RAPModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 24 | — | — | — | — | — | — | — | — | — | — | — | — | 41.82 | — | |
| LDBModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.02 | 24 | — | — | — | — | — | — | — | — | — | — | — | — | 43.37 | — | |
| CodexBackbone=Codex-12B, Filtering=Filtered, Shot=1-shot2021.07 | 22.78 | 25.1 | 24.52 | 27.15 | — | — | — | — | — | — | — | — | — | — | — | |
| Zero-shotModel Category=Base, Backbone=Qwen2.5-14B-Instruct, Evaluation Protocol=Zero-shot2026.02 | 18 | — | — | — | — | — | — | — | — | — | — | — | — | 38.02 | — | |
| SFTBackbone=Qwen-2.5-Coder, Parameters=7B2026.05 | 9.21 | — | — | — | — | — | — | — | — | 16.5 | — | — | — | — | — | |
| WITH GRPO ADVANTAGEBackbone=Qwen-2.5-Coder, Parameters=7B2026.05 | 7.72 | — | — | — | — | — | — | — | — | 14.7 | — | — | — | — | — | |
| ADVATNAGE WITH EXP()Backbone=Qwen-2.5-Coder, Parameters=7B2026.05 | 7.14 | — | — | — | — | — | — | — | — | 15.1 | — | — | — | — | — | |
| RLOOBackbone=Qwen-2.5-Coder, Parameters=7B2026.05 | 6.98 | — | — | — | — | — | — | — | — | 13.8 | — | — | — | — | — | |
| BASEBackbone=Qwen-2.5-Coder, Parameters=7B2026.05 | 4.5 | — | — | — | — | — | — | — | — | 11.9 | — | — | — | — | — | |
| CodexBackbone=Codex-12B, Filtering=Raw, Shot=1-shot2021.07 | 4.14 | 4.33 | 9.65 | 10.05 | 20.2 | 27.77 | 25.02 | — | — | — | — | — | — | — | — | |
| GPT-NeoBackbone=GPT-Neo 2.7B, Filtering=Raw, Shot=Zero-shot2021.07 | 3.9 | — | 5.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CPPOBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 54.6 | |
| CPPOBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 79.4 | |
| CPPOBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.6 | |
| DareFinetune Method=Dare, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 24.97 | 7 | — | — | |
| Direct SolveBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 42 | |
| Direct SolveBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 51.5 | |
| Direct SolveBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 69.6 | |
| DisenLoRAFinetune Method=DisenLoRA, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 27.11 | 9 | — | — | |
| LATS2025.02 | — | — | — | — | — | — | — | — | — | — | — | 69.46 | 50 | — | — | |
| LDB2025.02 | — | — | — | — | — | — | — | — | — | — | — | 60.64 | 40 | — | — | |
| MC-Tree-Of-Agents2025.02 | — | — | — | — | — | — | — | — | — | — | — | 79.72 | 64 | — | — | |
| MC-Tree-Of-AgentsStrategy=Pruning2025.02 | — | — | — | — | — | — | — | — | — | — | — | 85.18 | 76 | — | — | |
| MC-Tree-Of-AgentsStrategy=Refine2025.02 | — | — | — | — | — | — | — | — | — | — | — | 81.29 | 68 | — | — | |
| Pass@K Training / RLVRBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 47.6 | |
| Pass@K Training / RLVRBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60.7 | |
| Pass@K Training / RLVRBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.2 | |
| PKPOBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 44.8 | |
| PKPOBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 72.2 | |
| PKPOBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.7 | |
| Plan-and-SolveBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 49.2 | |
| Plan-and-SolveBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 53 | |
| Plan-and-SolveBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.1 | |
| PlanSearchBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 51.3 | |
| PlanSearchBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 55.4 | |
| PlanSearchBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 77 | |
| RAP2025.02 | — | — | — | — | — | — | — | — | — | — | — | 64.24 | 39 | — | — | |
| Reflexion2025.02 | — | — | — | — | — | — | — | — | — | — | — | 60.65 | 40 | — | — | |
| RethinkMCTS2025.02 | — | — | — | — | — | — | — | — | — | — | — | 76.6 | 59 | — | — | |
| SFT on allFinetune Method=SFT on all, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 22.55 | 7 | — | — | |
| SFT on cluster 0Finetune Method=SFT on cluster 0, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 20.67 | 6 | — | — | |
| SFT on cluster 1Finetune Method=SFT on cluster 1, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 21.22 | 4 | — | — | |
| SFT on cluster 2Finetune Method=SFT on cluster 2, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 16.65 | 7 | — | — | |
| SingleBackbone=GPT4omini2025.02 | — | — | — | — | — | — | — | — | — | — | — | 77.99 | 60 | — | — | |
| SingleBackbone=Claude2025.02 | — | — | — | — | — | — | — | — | — | — | — | 73.8 | 61 | — | — | |
| TiesFinetune Method=Ties, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 22.75 | 4 | — | — | |
| ToT2025.02 | — | — | — | — | — | — | — | — | — | — | — | 74.34 | 55 | — | — | |
| Tuple Planner SFTBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 47 | |
| Tuple Planner SFTBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 52.8 | |
| Tuple Planner SFTBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 71.3 | |
| TwinFinetune Method=Twin, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 19.1 | 5 | — | — | |
| UpSkillBase model=Qwen3.5-2B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 46.2 | |
| UpSkillBase model=Qwen3.5-4B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.1 | |
| UpSkillBase model=Qwen3.5-9B2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.2 | |
| w/o tuningFinetune Method=w/o tuning, Base Model=Meta-llama-3.1-instruct-8b2025.02 | — | — | — | — | — | — | — | — | — | — | — | 21.14 | 4 | — | — | |
| ZeroShot2025.02 | — | — | — | — | — | — | — | — | — | — | — | 56.56 | 35 | — | — |