Code Generation on HumanEval Python
75.6Pass@1INTERVENOR
Evaluation results
| Method | Links | ||
|---|---|---|---|
| INTERVENOR2023.11 | 75.6 | — | |
| Self-Collaboration2023.11 | 74.4 | — | |
| SFTBase Model=Qwen2.5-Coder-7B2026.06 | 74.3 | 82.2 | |
| Self-DebugVariant=Simple2023.11 | 73.8 | — | |
| Self-DebugVariant=UT+Trace2023.11 | 71.9 | — | |
| Self-PlayBase Model=Qwen2.5-Coder-7B2026.06 | 70.6 | 85.9 | |
| GPT-42023.11 | 67 | — | |
| CodeLLama2023.11 | 62.2 | — | |
| SFTBase Model=Qwen2.5-Coder-3B2026.06 | 62 | 75 | |
| PanGu-Coder22023.11 | 61.6 | — | |
| GPT-3.52023.11 | 60.3 | — | |
| TSP (OURS)Base Model=Qwen2.5-Coder-7B2026.06 | 58.9 | 76.9 | |
| WizardCoder2023.11 | 57.3 | — | |
| Self-PlayBase Model=Qwen2.5-Coder-3B2026.06 | 56 | 72.8 | |
| TSP (OURS)Base Model=Qwen2.5-Coder-3B2026.06 | 53.4 | 71.7 | |
| Base LLMBase Model=Qwen2.5-Coder-7B2026.06 | 51.6 | 71.5 | |
| Base LLMBase Model=Qwen2.5-Coder-3B2026.06 | 49.8 | 69.2 | |
| Claude2023.11 | 47.6 | — | |
| StarCoder2023.11 | 40.8 | — | |
| LoRAObjective=PEFT, Description=LoRA fine-tuning, Fine-tuning=MBPP, Configuration=Unmerged LoRA, α=16, r=82026.04 | 40.1 | — | |
| Various (merged from Round 1)Objective=Optimizer Comparison, Description=APPS fine-tuning, Fine-tuning=APPS (competition), Configuration=Various (merged from Round 1)2026.04 | 36.59 | — | |
| PaLM-Coder2023.11 | 35.9 | — | |
| TSP (OURS)Base Model=CodeLlama-7B2026.06 | 35.2 | 48.6 | |
| InstructCodeT5+2023.11 | 35 | — | |
| Base LLMBase Model=CodeLlama-7B2026.06 | 34.5 | 55.1 | |
| Code Llama-7B (vanilla)Objective=PEFT, Description=Baseline, Fine-tuning=None, Configuration=Code Llama-7B (vanilla)2026.04 | 34.15 | — | |
| Self-PlayBase Model=CodeLlama-7B2026.06 | 33.8 | 48 | |
| SFTBase Model=CodeLlama-7B2026.06 | 32.3 | 40.9 | |
| CodeT5+2023.11 | 30.9 | — | |
| LLama22023.11 | 30.5 | — | |
| CodeGeeX2023.11 | 22.9 | — | |
| CodeGen2023.11 | 18.3 | — | |
| InCoder2023.11 | 15.2 | — |