Code Generation on GSM8K Prolog (test)
89.3Pass@4Qwen2.5-Coder-7B + no KL divergence
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen2.5-Coder-7B + no KL divergenceConfiguration=One-shot, Steps=15002025.05 | 89.3 | |
| Qwen2.5-Coder-7B + no KL divergenceConfiguration=One-shot, Steps=10002025.05 | 88.3 | |
| Qwen2.5-Coder-7B + no KL divergenceConfiguration=Zero-shot, Steps=15002025.05 | 88.2 | |
| Qwen2.5-Coder-7B + no KL divergenceConfiguration=Zero-shot, Steps=10002025.05 | 88 | |
| Qwen2.5-Coder-7B + no KL divergenceConfiguration=One-shot, Steps=5002025.05 | 87.9 | |
| Qwen2.5-Coder-32BConfiguration=Zero-shot, Steps=Base2025.05 | 87.3 | |
| Qwen2.5-Coder-7B + length constraintConfiguration=One-shot, Steps=10002025.05 | 86.2 | |
| Qwen2.5-Coder-7B + length constraintConfiguration=One-shot, Steps=15002025.05 | 85.6 | |
| Qwen2.5-Coder-7B + length constraintConfiguration=One-shot, Steps=5002025.05 | 85.2 | |
| Qwen2.5-Coder-7B + no KL divergenceConfiguration=Zero-shot, Steps=5002025.05 | 84.7 | |
| Llama 3.3 70BConfiguration=Zero-shot, Steps=Base2025.05 | 83.9 | |
| Qwen2.5-Coder-7BConfiguration=One-shot, Steps=Base2025.05 | 81.4 | |
| Qwen2.5-Coder-7B + length constraintConfiguration=Zero-shot, Steps=10002025.05 | 78.2 | |
| Qwen2.5-Coder-7B + length constraintConfiguration=Zero-shot, Steps=15002025.05 | 78 | |
| Qwen2.5-Coder-7B + length constraintConfiguration=Zero-shot, Steps=5002025.05 | 76.4 | |
| Qwen2.5-Coder-7BConfiguration=One-shot, Steps=5002025.05 | 72.4 | |
| Mistral 7BConfiguration=Yang et al. (2024), Steps=Base2025.05 | 72 | |
| Qwen2.5-Coder-7BConfiguration=Zero-shot, Steps=5002025.05 | 71.4 | |
| Qwen2.5-Coder-7BConfiguration=One-shot, Steps=10002025.05 | 68.6 | |
| Qwen2.5-Coder-7BConfiguration=One-shot, Steps=15002025.05 | 67.2 | |
| Qwen2.5-Coder-7BConfiguration=Zero-shot, Steps=10002025.05 | 67 | |
| Qwen2.5-Coder-7BConfiguration=Zero-shot, Steps=15002025.05 | 66.7 | |
| Qwen2.5-Coder-7BConfiguration=Zero-shot, Steps=Base2025.05 | 63.7 |