List your company
ResearchFormerly WizWand

Code Generation on HumanEval+ (Pass Rate)

95.1Pass RatePass@10
0.77225.26149.7574.239Jul 2, 2025Aug 24, 2025Oct 16, 2025Dec 9, 2025Jan 31, 2026Mar 25, 2026May 18, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Pass@10Generator Model=gpt-oss-20b, Selection Pool Size (N)=102026.0495.1
Pass@10Generator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.0492.7
SEPGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.0489.6
Pass@10Generator Model=Qwen3-14B, Selection Pool Size (N)=102026.0489.6
GLM-4.7-FlashBase Model=GLM-4.7-Flash2026.0589
ZEDABase Model=Qwen3-30B-A3B2026.0588.5
ZEDASFTBase Model=GLM-4.7-Flash2026.0588.1
SEPGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.0487.8
NETSFT→OPDBase Model=Qwen3-30B-A3B2026.0587.4
SEPGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.0487.2
NETSFT→OPDBase Model=GLM-4.7-Flash2026.0587.1
CodeTGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.0486.8
ZEDASFTBase Model=Qwen3-30B-A3B2026.0586.8
NETSFTBase Model=GLM-4.7-Flash2026.0586.8
NETSFTBase Model=Qwen3-30B-A3B2026.0586.7
LLM-JudgeGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.0486.6
CodeTGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.0486.4
CodeTGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.0486.3
Dynamic SkippingBase Model=GLM-4.7-Flash2026.0586.3
ZEDABase Model=GLM-4.7-Flash2026.0586.3
Pass@10Generator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.0486
EmbeddingGenerator Model=Qwen3-14B, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0486
EmbeddingGenerator Model=gpt-oss-20b, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0486
Qwen3-30B-A3BBase Model=Qwen3-30B-A3B2026.0585.6
Pass@10Generator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.0485.4
EmbeddingGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0485.4
CodeBLEUGenerator Model=Qwen3-14B, Selection Pool Size (N)=102026.0485.4
Pass@1Generator Model=Qwen3-14B, Selection Pool Size (N)=12026.0484.5
CodeBLEUGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.0484.2
LLM-JudgeGenerator Model=Phi-4-reasoning, Selection Pool Size (N)=102026.0483.5
Pass@10Generator Model=Qwen3-4B, Selection Pool Size (N)=102026.0483.5
LLM-JudgeGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.0482.9
AdaMoEBase Model=GLM-4.7-Flash2026.0582.7
Pass@1Generator Model=Phi-4-reasoning, Selection Pool Size (N)=12026.0482.6
Pass@10Generator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.0482.3
SEPGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.0482.3
Pass@1Generator Model=gpt-oss-20b, Selection Pool Size (N)=12026.0482
CodeBLEUGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.0481.7
CodeBLEUGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.0481.1
EmbeddingGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0481.1
CodeTGenerator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=102026.0480.9
Pass@1Generator Model=Qwen3-4B-Instruct-2507, Selection Pool Size (N)=12026.0480.4
AdaMoEBase Model=Qwen3-30B-A3B2026.0580.3
SEPGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.0479.9
Pass@10Generator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.0478.7
CodeTGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.0478.6
LLM-JudgeGenerator Model=gpt-oss-20b, Selection Pool Size (N)=102026.0478
SEPGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.0476.8
CodeBLEUGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.0476.8
LLM-JudgeGenerator Model=Qwen3-4B, Selection Pool Size (N)=102026.0475.6
Pass@1Generator Model=Qwen3-4B, Selection Pool Size (N)=12026.0474.9
EmbeddingGenerator Model=Qwen3-4B, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0474.4
CodeTGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.0472.4
SEPGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.0472
CodeTGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.0471.6
Prefix-RFTBackbone=Qwen3-1.7B-Base2025.0768.9
CodeBLEUGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.0468.3
EmbeddingGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0468.3
LLM-JudgeGenerator Model=Phi-4-mini-instruct, Selection Pool Size (N)=102026.0467.7
SEPGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.0466.5
RFTBackbone=Qwen3-1.7B-Base2025.0764.9
Pass@1Generator Model=Phi-4-mini-instruct, Selection Pool Size (N)=12026.0464.7
CodeTGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.0463.5
EmbeddingGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0462.2
CodeBLEUGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.0460.4
Dynamic SkippingBase Model=Qwen3-30B-A3B2026.0559.1
Pass@1Generator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=12026.0458.9
LLM-JudgeGenerator Model=Llama-3.1-8B-Instruct, Selection Pool Size (N)=102026.0458.5
EmbeddingGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=10, Embedding Model=Qwen3-Embedding-0.6B2026.0457.9
CodeBLEUGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.0457.3
Pass@1Generator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=12026.0454.5
LLM-JudgeGenerator Model=Phi-4-mini-reasoning, Selection Pool Size (N)=102026.0454.3
SFT+RFTBackbone=Qwen3-1.7B-Base2025.0744.7
BaseBackbone=Qwen3-1.7B-Base2025.0737.8
SFTBackbone=Qwen3-1.7B-Base2025.074.4