List your company
ResearchFormerly WizWand

Code Generation on APPS Introductory

92.1pass@1LCP
0.37224.1864871.814Jul 7, 2021Apr 30, 2022Feb 22, 2023Dec 16, 2023Oct 9, 2024Aug 2, 2025May 27, 2026
Updated 5mo ago

Evaluation results

MethodLinks
LCPLLM backbone=GPT-4o2026.0392.1——————————————
LPWLLM backbone=GPT-4o2026.0387.2——————————————
LDBLLM backbone=GPT-4o2026.0378.7——————————————
BaselineLLM backbone=GPT-4o2026.0363.8——————————————
LogitsCoderModel Category=Decoding-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0238————————————62.21—
LATSModel Category=Search-guided Reasoning Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0235————————————53.21—
Contrastive DecodingModel Category=Decoding-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0235————————————56.23—
CODETModel=code-davinci-002, Setting=zero-shot, Temperature=0.8, Sampling number=502022.0734.6———————41.253.2—————
MCTSModel Category=Search-guided Reasoning Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0232————————————56.27—
Guided DecodingModel Category=Decoding-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0232————————————56.15—
self-playModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0231————————————51.46—
RethinkMCTSModel Category=Search-guided Reasoning Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0231————————————59.95—
BaselineModel=code-davinci-002, Setting=zero-shot, Temperature=0.8, Sampling number=502022.0727.2————————46.659.4————
ReflexionModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0227————————————45.38—
RAPModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0224————————————41.82—
LDBModel Category=Reflection-based Models, Backbone=Qwen2.5-14B-Instruct, Rollout Budget=202026.0224————————————43.37—
CodexBackbone=Codex-12B, Filtering=Filtered, Shot=1-shot2021.0722.7825.124.5227.15———————————
Zero-shotModel Category=Base, Backbone=Qwen2.5-14B-Instruct, Evaluation Protocol=Zero-shot2026.0218————————————38.02—
SFTBackbone=Qwen-2.5-Coder, Parameters=7B2026.059.21————————16.5—————
WITH GRPO ADVANTAGEBackbone=Qwen-2.5-Coder, Parameters=7B2026.057.72————————14.7—————
ADVATNAGE WITH EXP()Backbone=Qwen-2.5-Coder, Parameters=7B2026.057.14————————15.1—————
RLOOBackbone=Qwen-2.5-Coder, Parameters=7B2026.056.98————————13.8—————
BASEBackbone=Qwen-2.5-Coder, Parameters=7B2026.054.5————————11.9—————
CodexBackbone=Codex-12B, Filtering=Raw, Shot=1-shot2021.074.144.339.6510.0520.227.7725.02————————
GPT-NeoBackbone=GPT-Neo 2.7B, Filtering=Raw, Shot=Zero-shot2021.073.9—5.5————————————
CPPOBase model=Qwen3.5-2B2026.05——————————————54.6
CPPOBase model=Qwen3.5-4B2026.05——————————————79.4
CPPOBase model=Qwen3.5-9B2026.05——————————————82.6
DareFinetune Method=Dare, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————24.977——
Direct SolveBase model=Qwen3.5-2B2026.05——————————————42
Direct SolveBase model=Qwen3.5-4B2026.05——————————————51.5
Direct SolveBase model=Qwen3.5-9B2026.05——————————————69.6
DisenLoRAFinetune Method=DisenLoRA, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————27.119——
LATS2025.02———————————69.4650——
LDB2025.02———————————60.6440——
MC-Tree-Of-Agents2025.02———————————79.7264——
MC-Tree-Of-AgentsStrategy=Pruning2025.02———————————85.1876——
MC-Tree-Of-AgentsStrategy=Refine2025.02———————————81.2968——
Pass@K Training / RLVRBase model=Qwen3.5-2B2026.05——————————————47.6
Pass@K Training / RLVRBase model=Qwen3.5-4B2026.05——————————————60.7
Pass@K Training / RLVRBase model=Qwen3.5-9B2026.05——————————————76.2
PKPOBase model=Qwen3.5-2B2026.05——————————————44.8
PKPOBase model=Qwen3.5-4B2026.05——————————————72.2
PKPOBase model=Qwen3.5-9B2026.05——————————————78.7
Plan-and-SolveBase model=Qwen3.5-2B2026.05——————————————49.2
Plan-and-SolveBase model=Qwen3.5-4B2026.05——————————————53
Plan-and-SolveBase model=Qwen3.5-9B2026.05——————————————80.1
PlanSearchBase model=Qwen3.5-2B2026.05——————————————51.3
PlanSearchBase model=Qwen3.5-4B2026.05——————————————55.4
PlanSearchBase model=Qwen3.5-9B2026.05——————————————77
RAP2025.02———————————64.2439——
Reflexion2025.02———————————60.6540——
RethinkMCTS2025.02———————————76.659——
SFT on allFinetune Method=SFT on all, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————22.557——
SFT on cluster 0Finetune Method=SFT on cluster 0, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————20.676——
SFT on cluster 1Finetune Method=SFT on cluster 1, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————21.224——
SFT on cluster 2Finetune Method=SFT on cluster 2, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————16.657——
SingleBackbone=GPT4omini2025.02———————————77.9960——
SingleBackbone=Claude2025.02———————————73.861——
TiesFinetune Method=Ties, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————22.754——
ToT2025.02———————————74.3455——
Tuple Planner SFTBase model=Qwen3.5-2B2026.05——————————————47
Tuple Planner SFTBase model=Qwen3.5-4B2026.05——————————————52.8
Tuple Planner SFTBase model=Qwen3.5-9B2026.05——————————————71.3
TwinFinetune Method=Twin, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————19.15——
UpSkillBase model=Qwen3.5-2B2026.05——————————————46.2
UpSkillBase model=Qwen3.5-4B2026.05——————————————66.1
UpSkillBase model=Qwen3.5-9B2026.05——————————————76.2
w/o tuningFinetune Method=w/o tuning, Base Model=Meta-llama-3.1-instruct-8b2025.02———————————21.144——
ZeroShot2025.02———————————56.5635——