Code Reasoning on MBPP
84.7MBPP Execution AccuracyQwen2.5-Coder-CFB-Aug
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen2.5-Coder-CFB-Aug# Samples (K)=156, Base Model=Qwen2.5-Coder-7B-Base2024.10 | 84.7 | 69.3 | — | — | |
| Qwen2.5-Coder-ScaleQuest# Samples (K)=156, Base Model=Qwen2.5-Coder-7B-Base2024.10 | 83.1 | 69.9 | — | — | |
| Qwen2.5-Coder-CFB# Samples (K)=156, Base Model=Qwen2.5-Coder-7B-Base2024.10 | 77.2 | 64 | — | — | |
| DenserModel=GPT-4o2025.12 | 73.8 | — | — | — | |
| Process SupervisionModel=GPT-4o2025.12 | 73.2 | — | — | — | |
| Reflection-CoTModel=GPT-4o2025.12 | 72.6 | — | — | — | |
| Self-VerificationModel=GPT-4o2025.12 | 72.6 | — | — | — | |
| ReProBackbone=Qwen3-8B, Base Training=GRPO2025.12 | 72.1 | — | — | — | |
| Tree-of-ThoughtModel=GPT-4o2025.12 | 71.9 | — | — | — | |
| DenserModel=DeepSeek-V32025.12 | 71.7 | — | — | — | |
| Process SupervisionModel=DeepSeek-V32025.12 | 71.6 | — | — | — | |
| Self-ConsistencyModel=GPT-4o2025.12 | 71.2 | — | — | — | |
| Reflection-CoTModel=DeepSeek-V32025.12 | 71 | — | — | — | |
| Think-to-ThinkModel=GPT-4o2025.12 | 70.5 | — | — | — | |
| Self-VerificationModel=DeepSeek-V32025.12 | 70.4 | — | — | — | |
| Tree-of-ThoughtModel=DeepSeek-V32025.12 | 70.3 | — | — | — | |
| OriginalBackbone=Qwen3-8B2025.12 | 70 | — | — | — | |
| Self-ConsistencyModel=DeepSeek-V32025.12 | 69.6 | — | — | — | |
| Process SupervisionModel=o1-mini2025.12 | 69.5 | — | — | — | |
| DenserModel=o1-mini2025.12 | 69.5 | — | — | — | |
| Reflection-CoTModel=o1-mini2025.12 | 68.9 | — | — | — | |
| Think-to-ThinkModel=DeepSeek-V32025.12 | 68.9 | — | — | — | |
| GRPOBackbone=Qwen3-8B2025.12 | 68.8 | — | — | — | |
| Chain-of-ThoughtModel=GPT-4o2025.12 | 68.5 | — | — | — | |
| Self-VerificationModel=o1-mini2025.12 | 68.3 | — | — | — | |
| Tree-of-ThoughtModel=o1-mini2025.12 | 67.8 | — | — | — | |
| Self-ConsistencyModel=o1-mini2025.12 | 67.1 | — | — | — | |
| Chain-of-ThoughtModel=DeepSeek-V32025.12 | 66.9 | — | — | — | |
| Think-to-ThinkModel=o1-mini2025.12 | 66.4 | — | — | — | |
| Chain-of-ThoughtModel=o1-mini2025.12 | 64.7 | — | — | — | |
| ReProBackbone=Qwen3-4B-Base, Sample Count=Avg@4, Evaluation Domain=Out-of-domain, Base Method=GRPO2025.12 | 59.3 | — | — | — | |
| GRPOBackbone=Qwen3-4B-Base, Sample Count=Avg@4, Evaluation Domain=Out-of-domain2025.12 | 58.7 | — | — | — | |
| OursBackbone=DeepSeek-V2-Lite, Sparsity=50%2026.06 | 44.4 | — | — | — | |
| BaselineBackbone=DeepSeek-V2-Lite, Type=–2026.06 | 43.2 | — | — | — | |
| OursQBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%, Quantization=4-bit2026.06 | 38.5 | — | — | — | |
| OursQBackbone=DeepSeek-V2-Lite, Sparsity=25%, Quantization=4-bit2026.06 | 37.4 | — | — | — | |
| BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–2026.06 | 36.6 | — | — | — | |
| OursBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2026.06 | 36.6 | — | — | — | |
| OriginalBackbone=Qwen3-4B-Base, Sample Count=Avg@4, Evaluation Domain=Out-of-domain2025.12 | 25 | — | — | — | |
| CorDA# Facts=1002026.05 | — | — | 68 | 50.5 | |
| CorDA# Facts=2502026.05 | — | — | 66.8 | 32.32 | |
| CorDA# Facts=5002026.05 | — | — | 38.2 | 27.27 | |
| CorDA# Facts=10002026.05 | — | — | 56.3 | 12.12 | |
| LoRA# Facts=1002026.05 | — | — | 61 | 66.7 | |
| LoRA# Facts=2502026.05 | — | — | 63.2 | 66.7 | |
| LoRA# Facts=5002026.05 | — | — | 62.2 | 65.7 | |
| LoRA# Facts=10002026.05 | — | — | 62.6 | 64.6 | |
| OPLoRA# Facts=1002026.05 | — | — | 64 | 66.67 | |
| OPLoRA# Facts=2502026.05 | — | — | 64 | 65.66 | |
| OPLoRA# Facts=5002026.05 | — | — | 66.2 | 66.67 | |
| OPLoRA# Facts=10002026.05 | — | — | 62.3 | 67.68 | |
| PALoRA# Facts=1002026.05 | — | — | 59 | 67.7 | |
| PALoRA# Facts=2502026.05 | — | — | 60.8 | 68.7 | |
| PALoRA# Facts=5002026.05 | — | — | 58.2 | 67.7 | |
| PALoRA# Facts=10002026.05 | — | — | 61.7 | 68.7 |