Code Reasoning on CRUXEval
76.87AccuracyPPoT + Qwen2.5 Coder
Evaluation results
| Method | Links | |
|---|---|---|
| PPoT + Qwen2.5 CoderModel Size=7B, k (LLM samples)=5, m (PPoT samples)=52026.04 | 76.87 | |
| Qwen2.5 CoderModel Size=7B, k (LLM samples)=5, m (PPoT samples)=02026.04 | 73.37 | |
| Qwen2.5-Math-72B-InstructModel Series=Qwen2.5-Math, Size=72B2025.02 | 68.6 | |
| QwQ-32B-Preview*Model Series=QwQ, Size=32B, Source=Shen et al. (2025)2025.02 | 65.2 | |
| PPoT + Qwen2.5 CoderModel Size=3B, k (LLM samples)=5, m (PPoT samples)=52026.04 | 63.87 | |
| PPoT + Qwen2.5 CoderModel Size=7B, k (LLM samples)=1, m (PPoT samples)=52026.04 | 60.62 | |
| Llama-3.1-70B-Instruct*Model Series=Llama-3.1, Size=70B, Source=Shen et al. (2025)2025.02 | 59.6 | |
| Qwen2.5 CoderModel Size=3B, k (LLM samples)=5, m (PPoT samples)=02026.04 | 59.13 | |
| Qwen2.5 CoderModel Size=7B, k (LLM samples)=1, m (PPoT samples)=02026.04 | 57.12 | |
| PPoT + Qwen2.5 CoderModel Size=0.5B, k (LLM samples)=5, m (PPoT samples)=52026.04 | 53.13 | |
| DataFlow-Code-10KBase Model=Qwen2.5-14B-Instruct2025.12 | 52.9 | |
| Self-OSSBase Model=Qwen2.5-14B-Instruct2025.12 | 52.6 | |
| DataFlow-Code-5KBase Model=Qwen2.5-14B-Instruct2025.12 | 52.5 | |
| DataFlow-Code-1KBase Model=Qwen2.5-14B-Instruct2025.12 | 51 | |
| Qwen2.5-Math-7B-S2R-ORLModel Series=Qwen2.5-Math, Size=7B, RL Type=Outcome-level RL (ORL)2025.02 | 50.9 | |
| Code Alpaca-1KBase Model=Qwen2.5-14B-Instruct2025.12 | 50.2 | |
| Eurus-2-7B-PRIMEModel Series=Eurus-2, Size=7B2025.02 | 50 | |
| DataFlow-Code-10KBase Model=Qwen2.5-7B-Instruct2025.12 | 48.8 | |
| DataFlow-Code-5KBase Model=Qwen2.5-7B-Instruct2025.12 | 48.6 | |
| Qwen2.5-Math-7B-S2R-BIModel Series=Qwen2.5-Math, Size=7B, RL Type=Binary Reward (BI)2025.02 | 48 | |
| DataFlow-Code-1KBase Model=Qwen2.5-7B-Instruct2025.12 | 48 | |
| Qwen2.5-14B-InstructBase Model=Qwen2.5-14B-Instruct2025.12 | 48 | |
| Self-OSSBase Model=Qwen2.5-7B-Instruct2025.12 | 46.9 | |
| Qwen2.5 CoderModel Size=0.5B, k (LLM samples)=5, m (PPoT samples)=02026.04 | 46.13 | |
| PPoT + Qwen2.5 CoderModel Size=3B, k (LLM samples)=1, m (PPoT samples)=52026.04 | 45.87 | |
| Code Alpaca-1KBase Model=Qwen2.5-7B-Instruct2025.12 | 45.6 | |
| Qwen2.5-7B-InstructBase Model=Qwen2.5-7B-Instruct2025.12 | 44.8 | |
| Qwen2.5-Math-7BModel Series=Qwen2.5-Math, Size=7B2025.02 | 40.8 | |
| Qwen2.5 CoderModel Size=3B, k (LLM samples)=1, m (PPoT samples)=02026.04 | 40.5 | |
| PPoT + Qwen2.5 CoderModel Size=0.5B, k (LLM samples)=1, m (PPoT samples)=52026.04 | 37.37 | |
| AdamW2026.05 | 35.63 | |
| OpenMath2-Llama3.1-70B*Model Series=OpenMath2, Base=Llama-3.1-70B, Source=Shen et al. (2025)2025.02 | 35.1 | |
| MONA2026.05 | 35 | |
| Muon2026.05 | 33.62 | |
| Qwen2.5 CoderModel Size=0.5B, k (LLM samples)=1, m (PPoT samples)=02026.04 | 30.87 | |
| Qwen2.5-Math-7B-InstructModel Series=Qwen2.5-Math, Size=7B2025.02 | 28 |