Code Generation on HumanEval+ (Pass Rate)
95.1Pass RatePass@10
Evaluation Results
| Method | Links | |
|---|---|---|
| Pass@10Generator Model=gpt-oss-20b, Selection Pool Size (N)=102026.04 | 95.1 | |
| Pass@10Generator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.04 | 92.7 | |
| SEPGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.04 | 89.6 | |
| Pass@10Generator Model=Qwen3-14B, Selection Pool Size (N)=102026.04 | 89.6 | |
| GLM-4.7-FlashBase Model=GLM-4.7-Flash2026.05 | 89 | |
| ZEDABase Model=Qwen3-30B-A3B2026.05 | 88.5 | |
| ZEDASFTBase Model=GLM-4.7-Flash2026.05 | 88.1 | |
| SEPGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.04 | 87.8 | |
| NETSFT→OPDBase Model=Qwen3-30B-A3B2026.05 | 87.4 | |
| SEPGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.04 | 87.2 | |
| NETSFT→OPDBase Model=GLM-4.7-Flash2026.05 | 87.1 | |
| CodeTGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.04 | 86.8 | |
| ZEDASFTBase Model=Qwen3-30B-A3B2026.05 | 86.8 | |
| NETSFTBase Model=GLM-4.7-Flash2026.05 | 86.8 | |
| NETSFTBase Model=Qwen3-30B-A3B2026.05 | 86.7 | |
| LLM-JudgeGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.04 | 86.6 | |
| CodeTGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.04 | 86.4 | |
| CodeTGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.04 | 86.3 | |
| Dynamic SkippingBase Model=GLM-4.7-Flash2026.05 | 86.3 | |
| ZEDABase Model=GLM-4.7-Flash2026.05 | 86.3 | |
| Pass@10Generator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.04 | 86 | |
| EmbeddingGenerator Model=Qwen3-14B, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 86 | |
| EmbeddingGenerator Model=gpt-oss-20b, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 86 | |
| Qwen3-30B-A3BBase Model=Qwen3-30B-A3B2026.05 | 85.6 | |
| Pass@10Generator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.04 | 85.4 | |
| EmbeddingGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 85.4 | |
| CodeBLEUGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.04 | 85.4 | |
| Pass@1Generator Model=Qwen3-14B, Selection Pool Size (N)=12026.04 | 84.5 | |
| CodeBLEUGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.04 | 84.2 | |
| LLM-JudgeGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.04 | 83.5 | |
| Pass@10Generator Model=Qwen3-4B, Selection Pool Size (N)=102026.04 | 83.5 | |
| LLM-JudgeGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.04 | 82.9 | |
| AdaMoEBase Model=GLM-4.7-Flash2026.05 | 82.7 | |
| Pass@1Generator Model=Phi-4-reasoning, Selection Pool Size (N)=12026.04 | 82.6 | |
| Pass@10Generator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.04 | 82.3 | |
| SEPGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.04 | 82.3 | |
| Pass@1Generator Model=gpt-oss-20b, Selection Pool Size (N)=12026.04 | 82 | |
| CodeBLEUGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.04 | 81.7 | |
| CodeBLEUGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.04 | 81.1 | |
| EmbeddingGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 81.1 | |
| CodeTGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.04 | 80.9 | |
| Pass@1Generator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=12026.04 | 80.4 | |
| AdaMoEBase Model=Qwen3-30B-A3B2026.05 | 80.3 | |
| SEPGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.04 | 79.9 | |
| Pass@10Generator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.04 | 78.7 | |
| CodeTGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.04 | 78.6 | |
| LLM-JudgeGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.04 | 78 | |
| SEPGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.04 | 76.8 | |
| CodeBLEUGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.04 | 76.8 | |
| LLM-JudgeGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.04 | 75.6 | |
| Pass@1Generator Model=Qwen3-4B, Selection Pool Size (N)=12026.04 | 74.9 | |
| EmbeddingGenerator Model=Qwen3-4B, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 74.4 | |
| CodeTGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.04 | 72.4 | |
| SEPGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.04 | 72 | |
| CodeTGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.04 | 71.6 | |
| Prefix-RFTBackbone=Qwen3-1.7B-Base2025.07 | 68.9 | |
| CodeBLEUGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.04 | 68.3 | |
| EmbeddingGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 68.3 | |
| LLM-JudgeGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.04 | 67.7 | |
| SEPGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.04 | 66.5 | |
| RFTBackbone=Qwen3-1.7B-Base2025.07 | 64.9 | |
| Pass@1Generator Model=Phi-4-mini-instruct, Selection Pool Size (N)=12026.04 | 64.7 | |
| CodeTGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.04 | 63.5 | |
| EmbeddingGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 62.2 | |
| CodeBLEUGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.04 | 60.4 | |
| Dynamic SkippingBase Model=Qwen3-30B-A3B2026.05 | 59.1 | |
| Pass@1Generator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=12026.04 | 58.9 | |
| LLM-JudgeGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.04 | 58.5 | |
| EmbeddingGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.04 | 57.9 | |
| CodeBLEUGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.04 | 57.3 | |
| Pass@1Generator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=12026.04 | 54.5 | |
| LLM-JudgeGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.04 | 54.3 | |
| SFT+RFTBackbone=Qwen3-1.7B-Base2025.07 | 44.7 | |
| BaseBackbone=Qwen3-1.7B-Base2025.07 | 37.8 | |
| SFTBackbone=Qwen3-1.7B-Base2025.07 | 4.4 |