Code Generation on APPS
91.2Pass@1ExPairT-LLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ExPairT-LLMModel=OpenAI o1-mini2025.11 | 91.2 | — | |
| ExPairT-LLMModel=Codex2025.11 | 81.4 | — | |
| ExPairT-LLMModel=Gemini 2.5 Flash2025.11 | 70.3 | — | |
| SolvitaBackbone=Claude Opus 4.62026.05 | 69.3 | — | |
| Claude CodeBackbone=Claude Opus 4.62026.05 | 69 | — | |
| SolvitaBackbone=DeepSeek V4 Pro2026.05 | 68.1 | — | |
| SolvitaBackbone=GPT-5.42026.05 | 67.7 | — | |
| Codex CLIBackbone=GPT-5.42026.05 | 67.1 | — | |
| Codex CLIBackbone=DeepSeek V4 Pro2026.05 | 66.3 | — | |
| Claude CodeBackbone=DeepSeek V4 Pro2026.05 | 64.5 | — | |
| ExPairT-LLMModel=DeepSeek-R12025.11 | 64.3 | — | |
| Claude CodeBackbone=GPT-5.42026.05 | 60.4 | — | |
| Codex CLIBackbone=Claude Opus 4.62026.05 | 59.8 | — | |
| SolvitaBackbone=Grok2026.05 | 58.5 | — | |
| Codex CLIBackbone=Grok2026.05 | 57.2 | — | |
| OriginalModel=DeepSeek-R12025.11 | 56.3 | — | |
| AlphaCodiumBackbone=Claude Opus 4.62026.05 | 56.2 | — | |
| Claude CodeBackbone=Grok2026.05 | 55.5 | — | |
| MapCoderBackbone=Claude Opus 4.62026.05 | 55.1 | — | |
| SolvitaBackbone=Qwen3.62026.05 | 55.1 | — | |
| MapCoderBackbone=DeepSeek V4 Pro2026.05 | 54.8 | — | |
| OriginalModel=OpenAI o1-mini2025.11 | 54.7 | — | |
| B^4Model=Codex2025.11 | 54.3 | — | |
| MapCoderBackbone=GPT-5.42026.05 | 54.3 | — | |
| OriginalModel=Gemini 2.5 Flash2025.11 | 54.1 | — | |
| AlphaCodiumBackbone=DeepSeek V4 Pro2026.05 | 53.4 | — | |
| AlphaCodiumBackbone=GPT-5.42026.05 | 52.4 | — | |
| CODETModel=Codex2025.11 | 50.2 | — | |
| FlowSteerMethodology=Ours, Backbone=GPT-4o-mini2026.02 | 49.21 | — | |
| Codex CLIBackbone=Qwen3.62026.05 | 47.9 | — | |
| Claude CodeBackbone=Qwen3.62026.05 | 46.3 | — | |
| GPT-4oStrategy=Direct2024.10 | 46.2 | — | |
| AlphaCodiumBackbone=Grok2026.05 | 46.1 | — | |
| GPT-4oStrategy=MBTI2024.10 | 45.2 | -1 | |
| Orchestrator (Agent+RL)Methodology=Agent+RL, Backbone=GPT-4o-mini, Agent Configuration=Orchestrator2026.02 | 44.53 | — | |
| MapCoderBackbone=Grok2026.05 | 44.2 | — | |
| AFlow (4o-mini)Methodology=AFlow, Backbone=GPT-4o-mini2026.02 | 42.97 | — | |
| Router-R1 (Agent+RL)Methodology=Agent+RL, Backbone=GPT-4o-mini, Agent Configuration=Router-R12026.02 | 42.95 | — | |
| Single-passBackbone=DeepSeek V4 Pro2026.05 | 42.5 | — | |
| AlphaCodiumBackbone=Qwen3.62026.05 | 42.3 | — | |
| Agentflow (Agent+RL)Methodology=Agent+RL, Backbone=GPT-4o-mini, Agent Configuration=Agentflow2026.02 | 41.41 | — | |
| MapCoderBackbone=Qwen3.62026.05 | 41 | — | |
| Baseline (4o-mini)Methodology=Baseline, Backbone=GPT-4o-mini2026.02 | 40.62 | — | |
| Single-passBackbone=Claude Opus 4.62026.05 | 40.1 | — | |
| Baseline (Qwen3-8B)Methodology=Baseline, Backbone=Qwen3-8B2026.02 | 39.84 | — | |
| DeepSeek-Coder V2Strategy=Direct2024.10 | 39.4 | — | |
| GPT-5 OptimizerSource Model=GPT-4o, Target Model=o4-mini2025.12 | 38.44 | — | |
| PromptBridgeSource Model=GPT-4o, Target Model=o4-mini2025.12 | 38 | — | |
| Single-passBackbone=GPT-5.42026.05 | 37.9 | — | |
| Direct TransferSource Model=GPT-4o, Target Model=o4-mini2025.12 | 37.78 | — | |
| GEPASource Model=GPT-4o, Target Model=o4-mini2025.12 | 37.33 | — | |
| GPT-4o miniStrategy=MBTI2024.10 | 37.2 | 2.6 | |
| GPT-5 OptimizerSource Model=GPT-4o, Target Model=o32025.12 | 36.44 | — | |
| PromptBridgeSource Model=GPT-4o, Target Model=o32025.12 | 36.44 | — | |
| MIPROv2Source Model=GPT-4o, Target Model=o4-mini2025.12 | 35.33 | — | |
| GPT-4o miniStrategy=Direct2024.10 | 34.6 | — | |
| Single-passBackbone=Grok2026.05 | 34.6 | — | |
| DeepSeek-Coder V2Strategy=MBTI2024.10 | 34.4 | -5 | |
| GRPO (Qwen3-8B)Methodology=GRPO, Backbone=Qwen3-8B2026.02 | 34.38 | — | |
| GEPASource Model=GPT-4o, Target Model=o32025.12 | 32.89 | — | |
| Direct TransferSource Model=GPT-4o, Target Model=o32025.12 | 32.67 | — | |
| MIPROv2Source Model=GPT-4o, Target Model=o32025.12 | 32.67 | — | |
| Single-passBackbone=Qwen3.62026.05 | 30.8 | — | |
| SFT (Qwen3-8B)Methodology=SFT, Backbone=Qwen3-8B2026.02 | 26.56 | — | |
| OriginalModel=Codex2025.11 | 25.8 | — | |
| Llama3.1Strategy=MBTI2024.10 | 25.2 | 6.8 | |
| CodestralStrategy=MBTI2024.10 | 22.6 | 6.8 | |
| MapCoderLLM=GPT-42024.05 | 22 | — | |
| MapCoderLLM=GPT42025.02 | 22 | — | |
| CodeSimLLM=GPT42025.02 | 22 | — | |
| Llama3.1Strategy=Direct2024.10 | 18.4 | — | |
| Qwen-LongStrategy=MBTI2024.10 | 18.2 | 8 | |
| PromptBridgeSource Model=GPT-4o, Target Model=Gemma3-27B-it2025.12 | 17.92 | — | |
| PromptBridgeSource Model=GPT-4o, Target Model=Qwen3-32B2025.12 | 16.67 | — | |
| CodestralStrategy=Direct2024.10 | 15.8 | — | |
| GPT-5 OptimizerSource Model=GPT-4o, Target Model=Gemma3-27B-it2025.12 | 14.89 | — | |
| Self-PlanningLLM=GPT-42024.05 | 14.7 | — | |
| Self-PlanningLLM=GPT42025.02 | 14.7 | — | |
| Direct TransferSource Model=GPT-4o, Target Model=Qwen3-32B2025.12 | 14.45 | — | |
| DirectLLM=GPT-42024.05 | 12.7 | — | |
| DirectLLM=GPT42025.02 | 12.7 | — | |
| AnalogicalLLM=GPT-42024.05 | 12 | — | |
| GPT-4oModel Role=Source Model2025.12 | 12 | — | |
| GPT-4oSource Model=GPT-4o, Evaluation Context=Source Performance Reference2025.12 | 12 | — | |
| GPT-5 OptimizerSource Model=GPT-4o, Target Model=Qwen3-32B2025.12 | 12 | — | |
| CodeSimLLM=ChatGPT2025.02 | 12 | — | |
| AnalogicalLLM=GPT42025.02 | 12 | — | |
| MIPROv2Source Model=GPT-4o, Target Model=Llama3.1-70B-Instruct2025.12 | 11.33 | — | |
| MapCoderLLM=ChatGPT2024.05 | 11.3 | — | |
| CoTLLM=GPT-42024.05 | 11.3 | — | |
| MapCoderLLM=ChatGPT2025.02 | 11.3 | — | |
| CoTLLM=GPT42025.02 | 11.3 | — | |
| PromptBridgeSource Model=GPT-4o, Target Model=Llama3.1-70B-Instruct2025.12 | 11.11 | — | |
| GEPASource Model=GPT-4o, Target Model=Llama3.1-70B-Instruct2025.12 | 10.22 | — | |
| Qwen-LongStrategy=Direct2024.10 | 10.2 | — | |
| Direct TransferSource Model=GPT-4o, Target Model=Llama3.1-70B-Instruct2025.12 | 10 | — | |
| Direct TransferSource Model=GPT-4o, Target Model=Gemma3-27B-it2025.12 | 9.78 | — | |
| Self-PlanningLLM=ChatGPT2024.05 | 9.3 | — | |
| Self-PlanningLLM=ChatGPT2025.02 | 9.3 | — | |
| DirectLLM=ChatGPT2024.05 | 8 | — |