Code Synthesis on MBPP (pass@1, pass@80)
81.2Pass@1GTPO
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| GTPOBase Model=Qwen2.5-7B2025.11 | 81.2 | — | |
| GRPOBase Model=Qwen2.5-7B2025.11 | 81 | — | |
| Davinci CodexTraining stage=Code Finetuning, Shots=3-shot, Note=Calculation via OpenAI Codex API2022.04 | 50.4 | 84.4 | |
| PaLM-CoderTraining stage=Code Finetuning, Model scale=540B, Shots=3-shot2022.04 | 47 | 80.8 | |
| PaLMTraining stage=Pretraining only, Model scale=540B, Shots=3-shot2022.04 | 36.8 | 75 | |
| LaMDATraining stage=Pretraining only, Model scale=137B, Shots=3-shot2022.04 | 14.8 | 62.4 |