Code Generation on HumanEval OOD
98.44Pass@1SkillFlow
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SkillFlowMethod Type=Ours, Backbone Model=Qwen3.5-9B2026.05 | 98.44 | — | |
| v4-flashMethod Type=Baseline, Backbone Model=v4-flash2026.05 | 96.88 | — | |
| AgentFlowMethod Type=Agent+RL, Backbone Model=Qwen3.5-9B2026.05 | 93.75 | — | |
| FlowSteerMethod Type=Agent+RL, Backbone Model=Qwen3.5-9B2026.05 | 93.75 | — | |
| AFlowMethod Type=Search-based workflow, Backbone Model=Qwen3.5-9B2026.05 | 92.97 | — | |
| SkillRLMethod Type=Agent+RL, Backbone Model=Qwen3.5-9B2026.05 | 92.19 | — | |
| Qwen3.5Method Type=Baseline, Backbone Model=Qwen3.5-9B2026.05 | 89.06 | — | |
| Qwen3.5Method Type=GRPO, Backbone Model=Qwen3.5-9B2026.05 | 85.94 | — | |
| Qwen3.5Method Type=SFT, Backbone Model=Qwen3.5-9B2026.05 | 80.47 | — | |
| RPOBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 32.31 | — | |
| SAPO-iter1Backbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 31.09 | — | |
| SAPO-iter2Backbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 30.48 | — | |
| SAPO-iter3Backbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 30.48 | — | |
| SFT+GRPOBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 29.26 | — | |
| Online-RFTBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 28.04 | — | |
| RFTBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 26.82 | — | |
| CoTBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 25.6 | — | |
| SFTBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 25.6 | — | |
| SFT+GRPOBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 24.39 | — | |
| SAPO-iter3Backbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 24.39 | — | |
| SAPO-iter2Backbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 23.78 | — | |
| SAPO-iter1Backbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 22.56 | — | |
| RFTBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 21.34 | — | |
| SAPO-iter2Backbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 21.34 | — | |
| SAPO-iter3Backbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 21.34 | — | |
| SFTBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 21.34 | — | |
| RFT+DPOBackbone=Qwen-2.5-0.5B, Fine-tuning strategy=Full2026.01 | 20.73 | — | |
| Online-RFTBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 20.73 | — | |
| CoTBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 19.51 | — | |
| SFTBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 19.51 | — | |
| SAPO-iter1Backbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 18.9 | — | |
| RFTBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 18.9 | — | |
| RPOBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 18.29 | — | |
| RFT+DPOBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 17.68 | — | |
| CoTBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 17.07 | — | |
| RFT+DPOBackbone=Gemma-2-2B, Fine-tuning strategy=QLoRA2026.01 | 16.46 | — | |
| Online-RFTBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 15.85 | — | |
| RPOBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 15.24 | — | |
| SFT+GRPOBackbone=Llama-3.2-1B, Fine-tuning strategy=Full2026.01 | 13.41 | — | |
| Base ModelBase model=Qwen2.5-Math-7B2025.07 | — | 42.1 | |
| GRPOBase model=Qwen2.5-Math-7B2025.07 | — | 63.4 | |
| RL-PLUSBase model=Qwen2.5-Math-7B2025.07 | — | 68.3 | |
| SFTBase model=Qwen2.5-Math-7B2025.07 | — | 55.5 | |
| SFT+GRPOBase model=Qwen2.5-Math-7B2025.07 | — | 59.8 |