Code Generation on HumanEval+ (test)
98.1Pass@1SKETCHVERIFY (B: sem. vote)
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| SKETCHVERIFY (B: sem. vote)Model=Gemini 3.1 Pro, K=10, M=10, Thinking Level=low2026.05 | 98.1 | — | — | — | — | — | — | — | — | |
| GreedyModel=Gemini 3.1 Pro, K=10, M=10, Thinking Level=low2026.05 | 97.4 | — | — | — | — | — | — | — | — | |
| Majority VoteModel=Gemini 3.1 Pro, K=10, M=10, Thinking Level=low2026.05 | 97.4 | — | — | — | — | — | — | — | — | |
| Semantic Voting (flat)Model=Gemini 3.1 Pro, K=10, M=10, Thinking Level=low2026.05 | 97.4 | — | — | — | — | — | — | — | — | |
| Majority VoteModel=Gemini 3 Flash, K=10, M=10, Thinking Level=low2026.05 | 97 | — | — | — | — | — | — | — | — | |
| Semantic Voting (flat)Model=Gemini 3 Flash, K=10, M=10, Thinking Level=low2026.05 | 97 | — | — | — | — | — | — | — | — | |
| GreedyModel=Gemini 3 Flash, K=10, M=10, Thinking Level=low2026.05 | 96.3 | — | — | — | — | — | — | — | — | |
| SKETCHVERIFY (B: sem. vote)Model=Gemini 3 Flash, K=10, M=10, Thinking Level=low2026.05 | 96.3 | — | — | — | — | — | — | — | — | |
| Best-of-NModel=Gemini 3.1 Pro, K=10, M=10, Thinking Level=low2026.05 | 94.8 | — | — | — | — | — | — | — | — | |
| Best-of-NModel=Gemini 3 Flash, K=10, M=10, Thinking Level=low2026.05 | 93.9 | — | — | — | — | — | — | — | — | |
| Majority VoteModel=Gemini 3.1 Flash Lite, K=10, M=10, Thinking Level=low2026.05 | 92.7 | — | — | — | — | — | — | — | — | |
| Semantic Voting (flat)Model=Gemini 3.1 Flash Lite, K=10, M=10, Thinking Level=low2026.05 | 92.7 | — | — | — | — | — | — | — | — | |
| SKETCHVERIFY (B: sem. vote)Model=Gemini 3.1 Flash Lite, K=10, M=10, Thinking Level=low2026.05 | 92.1 | — | — | — | — | — | — | — | — | |
| GreedyModel=Gemini 3.1 Flash Lite, K=10, M=10, Thinking Level=low2026.05 | 85.4 | — | — | — | — | — | — | — | — | |
| ACES-OBackbone=Qwen2.5-Coder-14B2026.04 | 84.76 | — | — | — | — | 84.76 | 86.59 | — | — | |
| Majority VotingBackbone=Qwen2.5-Coder-14B2026.04 | 84.15 | — | — | — | — | 84.76 | 87.2 | — | — | |
| ACES-CBackbone=Qwen2.5-Coder-14B2026.04 | 84.15 | — | — | — | — | 84.15 | 86.59 | — | — | |
| GPT4-Turbo2023.12 | 81.7 | — | — | — | — | — | — | — | — | |
| Best-of-NModel=Gemini 3.1 Flash Lite, K=10, M=10, Thinking Level=low2026.05 | 81.7 | — | — | — | — | — | — | — | — | |
| ACES-OBackbone=Qwen2.5-Coder-7B2026.04 | 81.1 | — | — | — | — | 82.93 | 83.54 | — | — | |
| ACES-CBackbone=Qwen2.5-Coder-7B2026.04 | 79.88 | — | — | — | — | 82.32 | 82.93 | — | — | |
| Zero-shotBackbone=Qwen2.5-Coder-14B2026.04 | 79.4 | — | — | — | — | 82.97 | 85.96 | — | — | |
| Majority VotingBackbone=Qwen2.5-Coder-7B2026.04 | 78.66 | — | — | — | — | 82.32 | 82.93 | — | — | |
| ACES-CBackbone=DeepSeek-Coder-V2-16B2026.04 | 78.66 | — | — | — | — | 78.66 | 79.27 | — | — | |
| ACES-OBackbone=DeepSeek-Coder-V2-16B2026.04 | 76.22 | — | — | — | — | 76.22 | 78.66 | — | — | |
| Zero-shotBackbone=Qwen2.5-Coder-7B2026.04 | 75.63 | — | — | — | — | 81.38 | 85.62 | — | — | |
| MPSC-LexicalFoundation Model=GPT-3.5-Turbo2023.09 | 74.39 | — | — | — | — | 75 | 77.24 | — | — | |
| Majority VotingBackbone=DeepSeek-Coder-V2-16B2026.04 | 74.39 | — | — | — | — | 75.61 | 78.05 | — | — | |
| MPSC-SemanticFoundation Model=GPT-3.5-Turbo2023.09 | 73.54 | — | — | — | — | 74.46 | 75.26 | — | — | |
| MPSC-LabelFoundation Model=GPT-3.5-Turbo2023.09 | 73.47 | — | — | — | — | 76.66 | 77.25 | — | — | |
| TeacherModel=DeepSeek-Coder-6.7B2026.05 | 71.34 | — | — | — | — | — | — | — | — | |
| IF-CLSize=34B, Open-source=weight&data, Base Model=CODELLAMA2023.12 | 71.3 | — | — | — | — | — | — | — | — | |
| Zero-shotBackbone=DeepSeek-Coder-V2-16B2026.04 | 71.05 | — | — | — | — | 75.3 | 79.03 | — | — | |
| MT (DS-Coder-6.9B-Inst)Teacher model=DS-Coder-6.9B-Inst, Student model=N/A2026.04 | 70.7 | — | — | — | — | — | — | — | — | |
| GPT-42023.09 | 70.52 | — | — | — | — | 75.48 | 79.54 | — | — | |
| IF-CLPSize=13B, Open-source=weight&data, Base Model=CODELLAMA-PYTHON2023.12 | 69.5 | — | — | — | — | — | — | — | — | |
| IF-CLPSize=34B, Open-source=weight&data, Base Model=CODELLAMA-PYTHON2023.12 | 69.5 | — | — | — | — | — | — | — | — | |
| WeDLM-8BModel Type=WeDLM (Ours), Base Model=Qwen3-8B, Few-shot setting=4-shot2025.12 | 68.9 | — | — | — | — | — | — | — | — | |
| IF-CLSize=13B, Open-source=weight&data, Base Model=CODELLAMA2023.12 | 68.3 | — | — | — | — | — | — | — | — | |
| CodeTFoundation Model=GPT-3.5-Turbo2023.09 | 67.87 | — | — | — | — | 68.75 | 69.65 | — | — | |
| MagicoderS-CLPSize=7B, Open-source=weight&data2023.12 | 66.5 | — | — | — | — | — | — | — | — | |
| GPT3.5-Turbo2023.12 | 65.9 | — | — | — | — | — | — | — | — | |
| MPSC-UniformFoundation Model=GPT-3.5-Turbo2023.09 | 65.05 | — | — | — | — | 69.76 | 71.72 | — | — | |
| WizardCoder-CLPSize=34B, Open-source=weight2023.12 | 64.6 | — | — | — | — | — | — | — | — | |
| WeDLM-7BModel Type=WeDLM (Ours), Base Model=Qwen2.5-7B, Few-shot setting=4-shot2025.12 | 64 | — | — | — | — | — | — | — | — | |
| Self-consistencyFoundation Model=GPT-3.5-Turbo2023.09 | 63.5 | — | — | — | — | 64.7 | 65.67 | — | — | |
| Qwen3-8BModel Type=AR Baseline, Few-shot setting=4-shot2025.12 | 63.4 | — | — | — | — | — | — | — | — | |
| MBR-EXECFoundation Model=GPT-3.5-Turbo2023.09 | 62.12 | — | — | — | — | 67.08 | 71.38 | — | — | |
| GPT-3.5-Turbo2023.09 | 58.75 | — | — | — | — | 66.58 | 73.96 | — | — | |
| Qwen2.5-7BModel Type=AR Baseline, Few-shot setting=4-shot2025.12 | 53.05 | — | — | — | — | — | — | — | — | |
| LLaDA-1.5Sequence Length=512, Few-shot=02025.12 | 45.1 | — | — | — | — | — | — | — | — | |
| CODELLAMA-PYTHONSize=34B, Open-source=weight2023.12 | 42.7 | — | — | — | — | — | — | — | — | |
| ESPOSequence Length=512, Few-shot=02025.12 | 42.7 | — | — | — | — | — | — | — | — | |
| LLaDASequence Length=512, Few-shot=02025.12 | 41.5 | — | — | — | — | — | — | — | — | |
| WizardCoder-CLPSize=7B, Open-source=weight2023.12 | 40.9 | — | — | — | — | — | — | — | — | |
| VCRD-ProbTeacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 40.9 | — | — | — | — | — | — | — | — | |
| α-β divergence + Ours2026.05 | 38.41 | — | — | — | — | — | — | — | — | |
| DistilLLMTeacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 38.4 | — | — | — | — | — | — | — | — | |
| DistillLM-2Teacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 38.4 | — | — | — | — | — | — | — | — | |
| ABKDTeacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 37.8 | — | — | — | — | — | — | — | — | |
| diffu-GRPO(d1)Sequence Length=512, Few-shot=02025.12 | 37.2 | — | — | — | — | — | — | — | — | |
| KDTeacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 37.2 | — | — | — | — | — | — | — | — | |
| CODELLAMA-PYTHONSize=13B, Open-source=weight2023.12 | 36.6 | — | — | — | — | — | — | — | — | |
| ESPOSequence Length=256, Few-shot=02025.12 | 36.6 | — | — | — | — | — | — | — | — | |
| Forward KL + Ours2026.05 | 36.59 | — | — | — | — | — | — | — | — | |
| α-β divergence2026.05 | 36.59 | — | — | — | — | — | — | — | — | |
| Total Variation + Ours2026.05 | 36.16 | — | — | — | — | — | — | — | — | |
| SeqKDTeacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 36 | — | — | — | — | — | — | — | — | |
| GKDTeacher model=DS-Coder-6.9B-Inst, Student model=DS-Coder-1.3B2026.04 | 36 | — | — | — | — | — | — | — | — | |
| Skew FKL2026.05 | 35.98 | — | — | — | — | — | — | — | — | |
| Skew RKL2026.05 | 35.98 | — | — | — | — | — | — | — | — | |
| Skew RKL + Ours2026.05 | 35.98 | — | — | — | — | — | — | — | — | |
| Skew FKL + Ours2026.05 | 35.75 | — | — | — | — | — | — | — | — | |
| Total Variation2026.05 | 35.37 | — | — | — | — | — | — | — | — | |
| Reverse KL2026.05 | 34.15 | — | — | — | — | — | — | — | — | |
| Reverse KL + Ours2026.05 | 34.15 | — | — | — | — | — | — | — | — | |
| CODELLAMA-PYTHONSize=7B, Open-source=weight2023.12 | 34.1 | — | — | — | — | — | — | — | — | |
| Forward KL2026.05 | 33.54 | — | — | — | — | — | — | — | — | |
| Hard KD2026.05 | 32.93 | — | — | — | — | — | — | — | — | |
| wd1Sequence Length=512, Few-shot=02025.12 | 32.9 | — | — | — | — | — | — | — | — | |
| LLaDA-1.5Sequence Length=256, Few-shot=02025.12 | 32.3 | — | — | — | — | — | — | — | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=14B2026.02 | 30.5 | — | — | — | — | — | — | — | — | |
| LLaDASequence Length=256, Few-shot=02025.12 | 30.5 | — | — | — | — | — | — | — | — | |
| Proxy (Neural)CR=2.6, Model Size=14B2026.02 | 29.9 | — | — | — | — | — | — | — | — | |
| diffu-GRPO(d1)Sequence Length=256, Few-shot=02025.12 | 29.9 | — | — | — | — | — | — | — | — | |
| wd1Sequence Length=128, Few-shot=02025.12 | 29.9 | — | — | — | — | — | — | — | — | |
| wd1Sequence Length=256, Few-shot=02025.12 | 29.9 | — | — | — | — | — | — | — | — | |
| StarCoderSize=15B, Open-source=weight&data2023.12 | 29.3 | — | — | — | — | — | — | — | — | |
| Tokenizer-basedCR=3.7, Model Size=14B2026.02 | 29.3 | — | — | — | — | — | — | — | — | |
| MS (DS-Coder-1.3B)Teacher model=N/A, Student model=DS-Coder-1.3B2026.04 | 29.3 | — | — | — | — | — | — | — | — | |
| Tokenizer-basedCR=3.7, Model Size=7B2026.02 | 28.7 | — | — | — | — | — | — | — | — | |
| LLaDA-8BModel Type=DLLM Baseline, Few-shot setting=4-shot2025.12 | 28.05 | — | — | — | — | — | — | — | — | |
| Tokenizer-basedCR=3.7, Model Size=4B2026.02 | 28 | — | — | — | — | — | — | — | — | |
| Student (No Distill)Model=DeepSeek-Coder-1.3B2026.05 | 27.44 | — | — | — | — | — | — | — | — | |
| CodeGen-MonoSize=16B, Open-source=weight&data2023.12 | 27.4 | — | — | — | — | — | — | — | — | |
| Proxy (Neural)CR=2.6, Model Size=7B2026.02 | 26.8 | — | — | — | — | — | — | — | — | |
| CodeT5+Size=16B, Open-source=weight&data2023.12 | 26.2 | — | — | — | — | — | — | — | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=7B2026.02 | 26.2 | — | — | — | — | — | — | — | — | |
| Byte-levelCR=1, Model Size=14B2026.02 | 24.4 | — | — | — | — | — | — | — | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=4B2026.02 | 24.4 | — | — | — | — | — | — | — | — |